The Benefits of Label-Description Training for Zero-Shot Text Classification
Authors: Lingyu Gao
Debanjan Ghosh
Kevin Gimpel
Venue: EMNLP 2023 Main (2023)
Keywords: zero-shot
text classification
label description
TL;DR: We propose a simple way to improve model performance on zero-shot text classification by curating small label description finetuning datasets.

Abstract: Pretrained language models have improved zero-shot text classification by allowing the transfer of semantic knowledge from the training data in order to classify among specific label sets in downstream tasks. We propose a simple way to further improve zero-shot accuracies with minimal effort. We curate small finetuning datasets intended to describe the labels for a task. Unlike typical finetuning data, which has texts annotated with labels, our data simply describes the labels in language, e.g., using a few related terms, dictionary/encyclopedia entries, and short templates. Across a range of topic and sentiment datasets, our method is more accurate than zero-shot by 17-19% absolute. It is also more robust to choices required for zero-shot classification, such as patterns for prompting the model to classify and mappings from labels to tokens in the model's vocabulary. Furthermore, since our data merely describes the labels but does not use input texts, finetuning on it yields a model that performs strongly on multiple text domains for a given label set, even improving over few-shot out-of-domain classification in multiple settings.

The Benefits of Label-Description Training for Zero-Shot Text Classification
Lingyu Gao', Debanjan Ghosh and Kevin Gimpel"
'Toyota Technological Institute at Chicago
Educationnl Testing Service
(lygao, kgimpelJettic. edu, dghosh@ets.om org
Abstract
Label
Input
business
Pretrained language models have improved zero-shot text classification by allowing transfer of semantic knowledge from the train- ing data in order to classify among specific label sets downstream tasks. propose simple way further improve zero-shot accura- cies with minimal effort. We curate small fine- tuning datasets intended describe the labels for a task. Unlike typical finetuning data, which has texts annotated with labels, our data sim- ply describes the labels in language, e.g., using a few related terms, dictionary/encycloocdii entries, and short templates. Across range of topic and sentiment datasets, method is more accurate than zero-shot by 17-19% abso- lute. is also more robust to choices required for zero-shot classification, such as patterns for prompting the model to classify and mappings from labels tokens in the model's vocabulary. Furthermore, since our data merely describes the labels but does not use input texts, finetun- ing yields model that performs strongly on multiple text domains for given label set, even improving over few-shot out-of-domain classification in multiple settings.
finance
Business
Business the activity of making one's living making money by producing buying and selling products...
sports
racing
Sports
An athletic activity requiring skill physical prowess and often of competitive nature, racing, baseball...
(a) Topic classification
Label
Input
awful
Very
terrible.
Negative
A horrendous experience.
Very
great
Just fantastic.
Positive
Overall, it outstanding.
(b) Sentiment classification
Table 1: A few examples of LABELDESC training data for topic and sentiment classification.
the most probable verbalizer for the [MASK] posi- tion.! Although effective, the approach is sensitive to the choice of specific pattern/verbalizer pairs, with subtle changes in the pattern, the verbalizer, or both, often having a large impact on performance (van de Kar et al., 2022; Perez et al., 2021).
1 Introduction
Pretrained language models (PLMs) (Radford et al., 2018; Devlin al., 2019; Liu et al., 2019; Brown et al., 2020; Raffel et al., 2020) have produced strong results in zero-shot text classification for a range of topic and sentiment tasks, often using a putter-verbalizze approach (Schick and Schütze, 2021). With this approach, to classify the restaurant review "Overpriced, salty and overrated!". a pat- like "the restaurant is [MASK]" is appended to the review and verbalizers chosen for each label (e.g., "good" for positive sentiment and "bad" for negative). The text is classified by the pretrained masked language modeling (MLM) head to choose
To alleviate these issues, we propose a simple alternative approach of training on small curated datasets intended to describe the labels for task. Unlike typical training datasets, which consist of input texts annotated by hand with labels, data contains only the descriptions of the labels. We refer to this data as LABELDESE data and show a few examples for topic and sentiment classification in Table 1. For topic classification, we include a few terms related to the label (c.g.. "finance" for "Business", "racing" for "Sports"), definition of
Pleasee refer to Schick and Schlitze (2021) for more details on the nattern-verbalizer approach.
Co-senior authors.

---

the label from dictionary.com (e.g.. "An athletic activity ... for "Sports"), and a sentence from the opening paragraph of the label's Wikipedia article (e.g., "Business is the activity of..." for "Busi- ness"). For sentiment classification, we simply use related terms that capture the specific sentiment (e.g.. "terrible" for "Very Negative") as well as few hand-crafted templates (e.g., "It was t." where t is related term).
vised training set from one domain and testing on another (Section 4.2.4). In multiple cases, LA- BELTESCTRAINING actually attains higher accu- racy than fhw-shot supervised learning tested on out-of-domain test sets, even when hundreds of manually labeled training examples are used (al- beit from different input domain).
In summary, this paper shows several benefits of DEEEL.ESCTRAINEE
First, once a practitioner identifies label set of interest for zero-shot clas- sification, only requires a few minutes to collect the kind of LABELDESC data shown in Table 1, and training on this data improves over zero-shot by 17-19% absolute. Second, LABELDESCTRAADE ING leads to greater robustness to pattern/verbalize choice than zero-shot, Third, LABEL BESC data are domain independent with regard to the distribution of the inputs; a single LABELDESC training set can be used for any text classification task as long as contains the same labels. Our experiments show that this independence to input distribution leads to stable accuracy across domains, even attaining higher accuracy than out-of-domaii few-shot learn- ing on few cases.? ?
Next, we finetune pretrained models using the natter-verbaliie approach on LABELDESC data and evaluate them for text classification. For topic classification, we use patterns and verbalizers from Schick and Schütze (2022) to train on our LA- BELDESC examples by finetuning the model as well as the MLM head (see Section 3 for details). We refer to training on LABELDESC data as LA- BELDESCTRAINING In experiments, we show that
consistently improves accuracy (average improvement of 17-19%) over zero-shot classification across multiple topic and sentiment datasets (Table 2). We also show that LA- MELDESCTRAININ can decrease accuracy vari- ance across patterns compared to zero-shot classi- fication (Table 3), thus being less sensitive to the choice of pattern.
2 Tasks and LABELDESC Datasets
We evaluate on two types of tasks: topic classifi- cation on AGNews, Yahoo Answers, and DBPe- dia (Zhang et al., 2015) and sentiment classifica- tion on the Stanford Sentiment Treebank (SST) (Socher et al., 2013), Yelp Reviews (Zhang et al., 2015), IMDB (Maas et al., 2011), and Amazon Re- views Polarity (Zhang et al., 2015). We consider both binary and 5-way classification for SST and Yelp datasets (denoted as SST-2, SST-5, Yelp-2, and Yclp-5 henceforth) and only binary for IMDB and Amazon (denoted as IMDB and Amz-2 hence- forth). 3 Below we describe how we construct LA- BEL.DESC data for each label set. Dataset statistics as well as all LABELDESC data are in Section A.5 in the Appendix.
We then conduct additional experiments to re- veal the value of LABELDESCTRAINIL under various cireumstances. To study the impact of ver- balizer choice, we experiment with uninformative (randomly initialized) and adversarial (intentionally mismatched) verbalizers (Section 4.2.1). While ac- curacy drops slightly, both settings are still much more accurate than zero-shot classification with its original verbalizers. That is, LABELDESCTRAN ING is able to compensate for knowledge-free or even adversarial verbalizer choice. We also com- pare to finetuning randomly initialized classifier head without any patterns verbalizers, again finding accuracy be higher than zero-shot (Sec- tion 4.2.2). Collectively, our results demonstrate that LABEL DEECDDESTTAANNN leads to strong perfor- mance that less sensitive than zero-shot classifi- cation in terms of pattem/verbalizer choice, while also not requiring a pretrained MLM head.
Topic Classification. Since labels in topic clas- sification represent general concepts, we use both subjective descriptors of the labels (e.g., related terms) and objective sources of information (e.g.. dictionary definition and Wikipedia sentences)
Since LABELDESC data focuses entirely on the labels without seeking to capture the input text distribution, we would hope that it would ex- hibit stable performance across datasets with the same labels. So, we compare LABeLDESCTRAIN ING to the approach of training on a small super-
e Data and code are available https://gitthb com/ Langyugaa/Labeeeeserree
"Our method could be adopted for other tasks like natural language inference (NLI) using templates similar to how we approached sentiment classification. We leave a full explo- ration to future work.

---

when selecting LABELDESC data. In particular. we create LABELDESC examples for the label term itself, three related terms, a selected definition from dictionary.com, and the leading sentence from the label's Wikipedia article. As there are typically multiple dictionary.com definitions for our labels, we select single definition that best aligns with our understanding of the concept underlying the label. We use the leading Wikipedia sentence be- cause is typically brief overview/definition of the concept. Most labels in the Yahoo dataset con- sist of two keywords (c.g., Society & Culture). For these, we use both label terms, definitions for each, and the leading Wikipedia sentences for each.
nary sentiment (Yelp-2. SST-2, IMDB-2, Amz-2).
Hyperparameter Tuning. We adhere to the "true" zero-shot setting where hyperparameters can- not be tuned on a development set for the task of interest (Schick and Schütze, 2022). There- fore, we use a separate dataset for hyperparam- eter tuning the 20 Newsgroups (20NG, hence- forth) (Lang, 1995) a topic classification dataset with twenty labels. We select only four labels from 20NG for our purposes: talk.religoon.misc rec. rec.antos, sci.med, and palitissl We chose these four labels because they are sufficiently dis- tinct that we expect tuning to be informative for other real-world classification datasets; many of the other 20NG labels are highly technical or similar to one other, e.g., the pair comp.sys. mympp.ys.ibb.pp. parrdwaare and comp.sys. wwmmp..ys.mac.... mandchaadwareee as well as the pair comp. compp.o.mms-e www.oo....s.ii mas-windows.men misc and comp, winpwwindows.r We follow the same strategy as for topic classifi- cation above when constructing LABELDESC data for 20NG. The selected hyperparameters are used for both topic and sentiment elassifications.
We did not tune any of these decisions experi- mentally, so these choices in defining LEBELDESC data are almost certainly suboptimal. This subop- timality especially likely for the "World" label in the AGNews label set. This label reflects in- ternational news, but the dictionary definition and Wikipedia article for the term "World" do not cap- ture that sense of the word. Nonetheless, we did not change procedure for this label because wanted our results reflect real-world implemen- tation of the idea, complete with its limitations for certain labels.
3 Experimental Settings
The following settings are used in our experiments. Unless stated otherwise, we use the pretrained RoBERTa-base (b) and RoBERTa-large (l) mod- els (Liu al., 2019) for all experiments since RoBERTa the predominant choice in related zero- shot and dataless research (Schick and Schütze, 2021; van de Kar et al., 2022; Gera et al., 2022). Additionally, for every dataset, we use the entire available test sets for evaluation.
The LABELDESC instances we are using do not contain exhaustive information. We could easily extend the lists of related terms for each topic or use WordNet other semantic knowledge resources (Zhang et al., 2019). However, one of the goals of this research is to demonstrate how simple is to choose LABELDESC examples to improve zero-shot classification in very little time.
Sentiment Classification. We use a slightly dif- ferent procedure for sentiment classification. For 5-way sentiment, we use the label verbalizer itself write
Zero-shot Classification Baseline. We use the standard "pattern-verbalizer"" approach for topic and sentiment classification. The set of verbalizers used can be found in Table 10 in the Appendix. For choosing verbalizers, follow the choices of Schick and Schützze (2021) for AGNews, Yahoo, Yelp-5, and SST-5. We follow van de Kar et al. (2022) in choosing verbalizers for Yelp-2, SST-2, IMDB, and Amz-2 and we select verbalizers for DBPedia and 20NG ourselves.
and four synonym terms. In addition, four simple templates: "It was t.", "A(n) & experi- ence.", "Just t.", and "Overall, it was t.",, where is the label verbalizer or a synonym. For binary sentiment, we remove the neutral instances, com- bine the two positive labels ("Very Positive" and "Positive"" into one, and combine the two negative labels ("Very Negative" and "Negative") into one. This procedure produces& a total of 25 examples per label (5 terms 5 terms 4 templates) for 5- way sentiment and 50 examples per label for binary sentiment. Since these LABEI. instances are domain- inmeendindeeenddntt, we use the same data for both for 5-way sentiment (Yelp-5 and SST-5) and for bi-
Each pattern comprises a prompt including a [MASK] symbol placed before or after the text in- put, and aim to predict the masked token. For example, a prompt is added after the input a frame classification as a question answering task, e.g., "I Question: What is the topic of this news- group? Answer: [MASK]." We use RoBERTa-

---

Finetuning
Inference
Labels World
Verbalizers
Test data from AGNows
Sports
select verbalizers
2.Sports
"Need carbon sirik technologies Climate acientists conferince that greater efforts made to CO2 atmorphere.
3. Business
3. Busincas
finetune model prodict
4. SclfTech
Tech
correct
verbaltaor [MASK]
"Nead carbor technoloies Climate scientitss conference that greater efforts made to atmosphere
Sports
ingut gattem
Label
Quection tookc ariide7 Anguer (MASK)
Retated term:
Wikipedia Sport portains any form compeniivee plynical activity or...
create Input
2) "acing Quecioon What bORk artireed Anewer (MAS]
Model
3) "Sport to any foom compettive physical activity or... Quettion Ws topic this article" Anowen (MASK].
Dictionary: athletic activity rocuiring physical prowisss compestivee raciing
Sperta
Ruthen
Prediction: Sci/Tech
Figure I: Overview our proposed method, including the construction of LABELDESC data, the format of the text input, and the target used for both model finetuning and inference during test time. We present text inputs labeled "Sports" from the topic classification task, and use one of our patterns (see Table 11) here as an illustration. Note that all our LABELDESC datasets are balanced, with each pattern being associated with unique finetuned model checkpoint.
base/large with its MLM head for zero-shot experi- ments. Although the model is able to predict any token within its vocabulary, we choose only among the set of verbalizers, which are designed to be se- mantically coherent with class labels and tokenized into single token by the model's tokenizer.
quotes, and footers. We used batch size of and tuned over a set of five learning rates ((5e-7, le-6, 5e-6, le-5, 5e-5 }). Models were trained for 3500 training steps, evaluating on the dev set after each epoch, i.e., every 24 training steps since it's the size of LABELDESC dataset for 20NG, Based on tuning accuracies, chose learning rate 5e-7 and number of training steps 2160 for RoBERTa-base and 1920 for RoBERTa-large. Additionally, we explored variations of parameter freezing, such as freezing certain layers of RoBERTa. The best set- ting on 20NG was to freeze the lower half of the layers (excluding the embedding layer) during fine- tuning, used this for experiments reported below.
For topic classification tasks, use the PROMPT and Q&A patterns from Schick and Schütze (2022), which amounts to 14 patterns. For AGNews, we use "news/article" in the pattern tem- plates, while for Yahoo we replace this with "ques- tion", and for 20NG we use "newsgroup". For the sentiment classification tasks, create new Q& A patterns such as "r Question: What is the sentiment of this text? Answer: [MASK]." and PROMPT pat- terns such as "r Sentiment: [MASK]." where is the input text. There are 14 sentiment patterns in total, presented the Appendix (Section A.2).
Results and Analysis
In this section first present the results that are obtained via LABELDESCTRAINNS and then ana- lyze the benefits of LABEI.DESC data with a range of additional experiments and analysis.
We the same tings as the zero-shot baseline except that we fine- tune the models on LABELDESC data. We not use any target task data for tuning or early stopping. Instead, we fix hyperparameter values, including number of training steps, by tuning on 20NG fol- lowing the process described below.
LABELDESCTRAINE
4.1 Results
Table 2 compares standard zero-shot classifica- tion and LABELDESCTRAINING LESEI Training has higher accuracy across all topic and sentiment classification datasets, outperform- ing zero-shot by about 17% on average when using
We used LABELDESC data for the four selected 20NG labels our training data and the original 20NG data (training and test sets) as our dev set, re- stricted to the four selected labels shown Section 2. We preprocessed the data by removing headers,
"Section A.3 in the Appendix provides more details on hyperparameter tuning.

---

AGNews
Yahoo
DBPedia
Yelp
SST
Yelp
SST-2
Amzz
IMDB
Avg.
62.7
38.0
35.6
63.6
62.6
64.0
54.7
zero-shot
68.0
47.7
63.9
38.7
35.0
70.6
63.7
67.5
74.1
58.8
77.4
58.8
79.5
43.6
42.0
88.3
84.5
88.6
86.9
72.2
LABRLDESCTRAINNG
79.4
51.3
49.2
94.6
71.7
Chu al. (2021a) and Chu et al. (2021b) are dataless classification approaches (Chang et al., 2008) that include single-encoder and dual-encoder methods; the latter include the idea of embedding documents and labels and performing classifica- tion via semantic retrieval; we report their non- Table 2:
Test accuracy (%) comparison between zero-shot classification and DERHBDDDETTTRAINNNG RoBERTa For zero-shot, each result is the average over 14 patterns; and for LABELDESCTRAINING
base, = RoBERoeT-aaaceee each result the average over 14 patterns and three random seeds per pattern. The "Avg." column shows the average accuracies across columns.
AGNews
Yahoo
DBPedia
Yelp-5
SSTS
Yelp-2
SST-2
Amz-2
IMDB
b 7.4
7.0
18.9
4.3
4.3
10.7
11.0
10.3
13.2 17.0
zcro-shut
9.7
7.8
7.7
15.7
14.3
13.7
b
5.1.5.1,50
1.7 16
45,445,
2.0,
1.8.
2.1.
2.5, 1.9
13, 1.2, 1.4
1.8. 2.3, 1.8. 1.4
LDT
44,64
2.,
32,29,32
25,25,
16,
1.125.11 1.4
1.2,8, 1.6
09. 19,08
L. 1.2
Table 3: Standard deviations of test accuracy (%) across 14 patterns for each test dataset. For LABELDESCTRAIN- ING (LDT in the table), three random seeds were used so we show three standard deviations, one per random seed. All standard deviations over patterns are smaller for LDT than the corresponding values for zero-shot.
RoBERTa-base and 19% with RoBERTa-large. The results demonstrate that we can greatly improve the performance of zero-shot models with just few training examples that provide richer character- ization of the label but still without requiring any textual inputs from the task datasets.
ensemble results in Table 4. Schick and Schütze (2022) use labeled training data (10 or 100 exam- ples, see Table 4) for each task, which differs from the domain- ienmpnnnnddeenddett LABELDESC examples which agnostic the domain of the textual in- puts. From van de Kar et al. (2022), we include the highest accuracies.
Table 3 shows that accuracy variances across patterns using DABELDESCTRAINIE
are much lower than the zero-shot setting, which is known to be unstable (Perez al., 2021). Finetuning LABELDESC data not only improves accuracy, but also mitigates sensitivity to pattern selection.
The results of AABRI.DESCTRAINIE are com- parable other methods across datasets. For sen- timent classification, LABELDESCTRAINNS per- forms better than dataless classification (Chu et al., 2021a) by a large margin for all datasets and is com- petitive with van de Kar et al. (2022) and Schick and Schütze (2021). Our method is better than that of van de Kar et al. on topic datasets (AGNews, Yahoo, and DBPedia) but not sentiment datasets cept for SST-2. van de Kar et al. (2022) search for naturally occurring data large corpora; texts ex- pressing sentiment are well-represented corpora, while texts for topics in a fixed label set may be rarer. DABEL.DESCTRAINEE AINING trains on balanced data from a fixed label set, leveraging available knowledge resources to inform about topics.
Comparisons to the State of the Art. We com- pare to sfte-of-the--r (SOTA) results from the lit- erature in Table (we show results using RoBERTa- base to better compare to other methods). For this comparison, we use only single pattern with LA- BELDESCTRAINNNG since doing so reflects more of a real-world use case than averaging over 14 patterns. We choose a single pattern for each of RoBERTa-base and large by tuning on 20NG as we did for other hyperparameters. We use three random seeds and report average accuracies and standard deviations over seeds.
Although van de Kar et al. (2022) do not report 5-way classification results for Yelp or SST, we report results for both datasets (including base and large models) so that future work can compare to our results in this table. We recommend tuning zero-shot and few-shot methods on datasets that
"We only include results with PROMPT and Q&A patterns (14 patterns for topic and 16 for sentiment) from Schick and Schlitze (2022), since those are the pattern types we used for LABELDESCT RESCTRAINNGG.
Please refer to A.3 and Table 14 in Appendix for details. We use the same setting for Table 5.

---

AGNews
Yahoo
DBPedia
Yelp
Yelp
SST.5
SST-2
Amz
IMDB
84.610..
82.411.2
42.01:44
848810
44.31:01
88.2.002
89.6104
03
TRAINNNGG
1..1
61.2=2a
$8.5
52.5+12
95.3-b04
1.
91.4:08
94.5
92.9=0.1
Chu et. (2021a)
6
68.8
57.8
81.9
67.3
65.0
66.8
Cha et (20211)
b
75.1
00.0
88.6
10
79.5+
58.4=27
44.3=225
Schick Schüize (2022)
100
87.5
65.3=110
54.8
van de Kar et al. (2022)
79.2
56.1
80.4
92.0
85.6
92.0
86.7
Table 4: Test accuracy (%) comparison to state-of-the-art methods. 10/100 =# laheled examples used.
AGNows
Yahoo
DBPedia
Xelp
Yelp-2
SST-5
SST-2
Amz-2
IMDB
LARRI DESCTRAINING
84.3.10.1
57.542077 57.5
8..011.5
41 11.62:
83.1
35.3..
86.71:066
90.8100
83.1
85.5
88.1
53.8+199
95.4
51.4+:1.
90.3+
94.2+03
94.1
text-davinci--e (acro-shot) text-davinci-O00 (ICI.)
80.2
58.5
47.2
92.3
93.3
78.9
83.9
61.1
84.2
57.0
92.9
51.2
92.3
95.1
88.3
Table 5: Test accuracy (%) comparison to text-davinci-003 on test set subsets.
are excluded from the final comparison, like 20NG in this paper.
pairs (Schick and Schütze, 2021, 2022). Here, we study several variations of LEABE..ESCTRAANE to investigate whether we can simplify remove components of these pattern- verbalizeerr pairs. We first experiment with changing verbalizers to gauge the impact of verbalizer choice for LABELDESC TrainIng (Section 4.2.1). Next, we conduct clas- sification experiments that do not use patterns or verbalizers at all (Section 4.2.2).
Comparisons Involving GPT-3.5. Our method not only works for MLMMstyle models like RoBERTa, but also for autoregressive models. In Table 5, we show zero-shot and in-context learning (ICL), where we use the entire LA- BELDESC data for the task as ICL demonstra- tions, with text-davinci-003 (GPT-3.5; OpenAl, 2022). Due to our restricted budget, we decided to use only 1,000 test instances for each test dataset in GPT-3.5 experiments, while ensuring that the label distribution remains consistent with that of the full test dataset. It is well known that ICL is sensitive variety of design choices, including the order of the demonstrations (Fei al., 2023; Lu al., 2022). For ICL demonstrations, we included all LABELDESC data for a task to make predictions for each test instance. To avoid the "recency bias" (i.e., the tendency to predict labels that occur to- wards the end of the prompt; Zhao et al., 2021 a), randomly shuffle the order of demonstrations. We left other parameters untouched. GPT-3.5 with ICL using LABELDESC data outperforms zero-shot GPT-3.5 on all datasets, showing the value of LA- BELDESC data even in-domain inputs are un- available. In comparison to GPT-3.5 flavors, LA- BELDESCTRAINNG (RoBERTa-large) performs better on AGNews, DBPedia, Yelp-2, SST-5, and IMDB, and is competitive across other datasets.
Furthermore, include more baseline, i.e., the model finetuned on the 20NG LABELDESC data and patterns analyze the generalizability (Section 4.2.3). We also report additional experi- ments in which we measure the multi-domain ro- bustness of DESCIEESCTAAAEE standard procedure of training on one domain and testing out-of-domain test set (Section 4.2.4), Finally, we take closer look label-wise per- formance to better understand how LABELDE- SCTRAINING outperforms zero-shot classification (Section 4.2.5).
compared to a
4.2.1 Impact of Verbalizers
In this section we report experiments with LA- without meaningful verbaliz- and even with adversarially chosen verbalizers. We explore two different verbalizer settings:
RANDOM: We add c words, i.e., RAN- DOMI, RANDOM2, RANDOMe, where e is the number dataset labels, to the model's 's vocabulary and randomly initialize their embed- dings. This setting prevents the use of any prior knowledge in the verbalizer embeddings.
4.2 Analysis and Discussion
One of the primary requirements of the zero-shot approach is the availability of vettern-verbalzer
MISMATCHED: We shuffle the original mapping

---

AGNews
Yahoo
DBPedia
SST-5
Yelp 2
SST-2
Amz
IMDB
Avg.
62.71:744
41.5.170
54.61189
38011:3
35.6114.
63 10.7
62.61:11.0
64:01:00.
69.91132
54.71997
nero-shot
68.0 88.0
47.7
63.9199
38.7778
35.0:777
70.61:157 15.7
63.72:14.
67.5
74.1
58.8
61.8+700
49.4+552
12.9478
34,6+446
36,5+3377
67,74-10.3
63.4+997
67.2++99
12.5+1005
58.447,6
LDT2NG
72.41668
44.4144
11.91108
36.3257
36.6177
63.411130
56.9287
6091
67.51152
57.819.1
LDT
77.4+:9
18
79.54:44
43.6
1.
88.3+255
84.5#22
88.61:14
86.9
72.242.5
5.0
60.8:2.1
8666130
51.3224
19.2
94.6
91.3
94.12:13
92.1
77.7
b
77 31:40
$4.3139
81 3173133
98 181133
37.02332
7N 78.4-1:00 10.0
73.3:279
80.02:9
73.8199
65.91666
MLM.
75.21:50
58.0130
15.41130
46.4133
43.4129
90.81776
84.11:68
90:2271
87 874
73.4116.1
MLM
73.1+:56
50.1+54
72.6+-8.1
368822
80.1+772
75.8+:5.0
81.86688
76.7=60
64.8+5.5
66.4::..6
44.5:449
73.1
41.92400
38.7:44.2
83.6:6655
78.1
85.0
77.7
65.4:6..0
classifier
b
72.5 5
57.1
87.7426
40.32:3
39.42225
86.9122
79.72:1.1
19:09::99
80662:36
70.4113
7..8+11
50,9+7.3
78.2+100
42.4+16
35.3+9..2
93.3++0.9
86.6+1144
93.7+005
85.7+20
71.54288
Table 6: Test accuracies (%) for several variations of AABELDESCTTRINNNE The standard deviations are computed over 14 patterns for zero-shot; 3 random seeds for the classifier (no patterns); and both 14 patterns and 3 random seeds for LABELDESCTRAINII on 20NG, LABELDESCTTRAINDEE RANDOM, and MISMATCHED (LDT2ONG. LDT, MLM,, and MLM,, in Table).
of labels to verbalizers, ensuring that each verbal- izer maps to a different label than in the original LABELDESCTRAIINGG setting. Since we are still finetuning the embeddings, finetuning can help the model recover from this mismatched initialization.
using a classifier that has been finetuned on a small LABEL. DESC dataset may serve as a strong alterna- tive to the pattein-verbalizer approach.
4.2.3 Cross-Task Generalizability
We report results on the model finetuned on the 20NG LABELDESC data and patterns, i.c., LA- BEL.DESCTAINNGG on 20NG (LDT20NG), in Ta- ble 6. While the patterns for the reported datasets different from those used for 20NG, especially for sentiment datasets, they have similar structures (see Section A.2). For RoBERTa-base, 2DT often outperforms zero-shot results, except for AG- News and Yelp-5. However, for RoBERTa-large. while 2DT outperforms the zero-shot results on all topic classification datasets, it's worse on sentiment classification except for SST-5,
The results are shown in Table 6. Since we still use the MLM head for these results, we refer to them as "MLM, RANDOM" and "MLM, MIS- MATCHED". While LABEL.DESCTRAINNS per- forms better than RANDOM, and RANDOM is bet- ter than MISMATCHED, both are better than zero- shot on average. These results suggest that LA- BELDESC data can partially compensate when the quality of the verbalizers is unknown or poor, at least to improve over zero-shot.
4.2.2 Classifiers Without Patterns or Verbalizers
4.2.4 Multi-Domain Evaluation
Since finetuning on LEBEIDESC data outperforms zero-shot results with RANDOM verbalizers. we also evaluate its performance without patterns, i.e., using a standard randomly initialized softmax clas- sifier. The input is the original text without any patterns and we use a two-layer classification head on top of the [CLS] token representation of the pretrained models.
Since LASEL examples are domain- independent, they can be used for multiple datasets that have the same labels. To assess the multi- domain performance of LABELDESCTRAINING we compare it to supervised few-shot learning in which a model is trained on data from one domain and then evaluated on a different domain with the same label set (i.e., training on SST-5 and evaluating on Yelp-5). To create multi multi--omainn test sets for a single topic label set, we keep AGNews as it is and create new subsampled version of Yahoo follows: (1) "Polities & Government" and "Socicty & Culture" texts are assigned the label "World", (2) "Sports" texts are labeled "Sports", (3) "Business & Finance" texts are labeled "Business", and (4) "Science & Mathematics" and "Computers
The bottom two rows of Table 6 show the re- sults. The classifiers are close to that of the MLM/RANDOM setting and still much higher than zero-shot on average, suggesting that it is not nec- essary use patterns, verbalizers, even the pre- trained ML.M head in order to outperform zero-shot classifiers. If is difficult to select verbalizers or design patterns for a particular classification task,

---

& Internet" texts are labeled "Sci/Tech". Other Yahoo texts are removed. We refer this new version of the Yahoo dataset as YahoOaG- For sentiment classification, we choose two dataset pairs that share label sets, i.e., SST-5 and Yelp-5.
AGNews test set
Yahoo test set
test accunec (9))
80
80
70
60
We do not change anything about the LABELDE- sctraIning configuration for these experiments. We simply evaluate the same model on multiple test sets, reporting average accuracies over patterns.
10
500
100
600
Velp-5 test set
SST-5 lest
aocoracc
For few-shot setup, we create datasets with 10, 100, and 500 training examples per label. For in-
60
domain experiments, train, dev, and test sets are drawn from the same domain/dataset, whereas for out-of-domain experiments, train and dev sets are drawn from one domain and the test set is drawn from another domain. We tune learning rates over the same ranges as mentioned carlier and use batch sizes 1, 2, and 4 for 10, 100, and 500 examples per label, respectively. We train for 15 epochs and select the checkpoint from the best epoch selected by the dev set.
10 100
500
10
100
000
in-domain
out-of-domain
LDT
Figure 2: Domain transfer results, where the X-axis
shows the number of training examples per label.
"Neutral" are very low for the zero-shot models SST-5, indicating that those labels are being largely ignored. Again, LABELDESCTRANDE large improvements in for some of these labels, especially "Very Positive". These trends are likely due in part to the differences verbalizer probabili- ties, e.g., "good" and "bad" occur more frequently than "great" and "terrible". The LEBELDESC data is balanced, which helps to mitigate the ignoring of labels, even though the task test sets anced. Table 7 shows examples that are incorrectly classified by zero-shot models but are correctly classified by the LABELDESCTRAININ models.
shows
The results using RoBERTa-large are shown in Figure 2. For brevity, we only show a subset of results.? 7 As we would expect, testing on out-of- domain data lcads to accuracy drops but adding more out- -fttof-- training data reduces this gap. LABELDESCTRAINIEE
shown an orange dot- ted line, outperforms supervised few-shot learning in some cases, such training on AGNews and testing on YahooAG, even with 500 examples per la- bel (upper-right plot in Figure 2). We see the same trend when the supervised model is trained on Yclp- 5 and tested on SST-5 (lower-right plot in Figure 2). outper-
all bal-
5 Related Work
In 3 out of 4 cases, MLABELDESCTRARO forms supervised few-shot out-of-doman learning with 10 examples per label, outperforming 100 in 2 out of 4 cases.
One common approach in zero-shot text classifi- cation is to transfer knowledge from seen labels (Dauphin et al., 2014), which requires observed la- bels and a notion of label similarity. Some sources of semantic knowledge used for this purpose in- clude multiple modalities (Lampert al., 2009), la- bel relationships in knowledge graphs (Wang al., 2018), 8), and word representations (Song and Roth, 2014; Fei et al., 2022).
4.2.5 Label-wise Investigation
To better understand why LABELDESCTRAINNNG outperforms zero-shot, we report label-specific Fl scores in Tables 8 and 9. For AGNews, the zero- shot classifiers have low Fl scores for the World label. probably because the verbalizer "World" is much less coherent and less representative of the actual label than others like "Sports." LA scTraininG improves Fl on the World label by roughly 20 points, while the improvement for Sports is only about 4 points. Likewise, the Fl scores for "Very Negative", "Very Positive", and
There are several other approaches to zero-shot classification. To classify documents, Chang et al. (2008) used knowledge-based text representations derived from Wikipedia, and Barak al. (2009) used both Wikipedia and WordNet. Zhang et al. (2019) combined label descriptions with label hierarchy and word-to-label paths in ConceptNet, with data augmentation strategies. Yin et al. (2019) used a textual entailment approach with label defi-
"Section A.4 in the Appendix shows additional results.

---

text (Iheadline (atedddlieee/exxxtteeee body] for AGNews)
zero-shot
LABELDESCTRAINING
[Homeless families total [Te figure for homeless families in England has Business
World
lopped 100,000 for the first time.]
[Shifting signs in North [Kre Jong Il dials back his personality cult Sports protest activities pick up-]
World
[GM. Daimler Go in both firms' portfolios.]
will help the companies compete and fill gaps
Sci/Tech
Business
(u)reeentingll stupid.
Positive
Very Negative
Still, I'm not quite sure what the point is...
Positive Positive
Negative
This 72-minute film does have exciting scenes, but it's tad slow.
Neutral
Table AGNews/SST-5 data that are correctly classified with LABBLDESCTRAINGE but not zero-shot settings.
zero-shot LABeLDESCTRAINING
less classification (Chang et al., 2008) which in- volves building classifiers by designing or learning generic function that scores the compatibility of document and label defined in natural language. We compared empirically to the dataless classifi- cation approaches of Chu al. (2021a) and Chu et al. (2021b) who used pretrained models, natu- rally annotated data like that from Wikipedia cat- egories, and unsupervised clustering techniques. There is a wealth of prior work in semi-supervised text classification (Nigam et al., 2000; Xie et al., 2020; Howard and Ruder, 2018). There is also lated work generating label names (Schick al., 2020) label descriptions (Chai al., 2020; Sun et al., 2019) but for supervised text classification.
World
61.5+151
81,0=43
Business
63.6=71
74,9=47
Sports
88.2=39
92.7245
Sci/Tech
55.0+114
67.8=93
Table 8: AGNews label-wise Fl (RoBERTa-large).
zero-shot
LABEIDESCTRAINIGG
Very Negative
11.2114
25.8257
Negative
37.6+212
62.5#20
Neutral
1.2129
10,8
Positive
46.0-5a
48.2=49
Very Positive
12.1
58.0=40
Table
SST-5 label-wise Fl (RorERTa-large).
nitions from WordNet. Another approach that has gained popularity is self-training given label names and mining an unlabeled dataset (Meng et al., 2020; Gera et al., 2022). van de Kar et al. (2022) extend the mining-basse based approach by selecting unsuper- vised examples (via patterns) for training. Basile al. (2022) select label descriptions by aggrega- tion. Meng et al. (2022) use language models to generate new training examples. On the contrary, we train on a small set of domain- iempenennddeennennt la- bel deseriptions. Our setup is influenced by Schick and Schütze (2021, 2022), although, instead of finetuning on training examples, we only use our LABELDESC data.
Conclusions
We presented LABELDESCTRAINING a method for improving the accuracy of zero-shot classifica- tion by using small, curated datasets that simply de- scribe the labels for a task in natural language. Our method is 17-19% more accurate than zero-shot on average across a range of datasets. DF SCTRANNIG is also more robust to the choices required for zero-shot classification, such pat- terns and verbalizers. Furthermore, LABELDESC data is domain agnostic and therefore can used for any text classification task as long as contains the same set of labels. LABEL.DESCTE DRAEEEESTTTTTNCC AINING can even outperform a supervised approach that uses training data from different domain. One future direction would be to apply the idea to structured prediction, NLI, and natural language generation tasks. Another would be to investigate ways reduce the dependence of pretrained models on pat- terns and verbalizers, such as directly calibrating the marginal probabilities of verbalizers with the goal of minimizing biases of pretrained models.
Autoregressive language models have also been used for zero-shot text classification; we report zero-shot and ICL results with LABELDESC data using GPT-3.5 (OpenAI, 2022). Zhao et al. (202 Ib) found it beneficial to "calibrate" such models for this setting; this idea is not immediately applica- ble here due our use of encoder-only models like RoBERTa. Calibration could be extended to encoder-only models, which we plan to explore in future work. Our work is closely related to data-

---

7 Limitations
pers, pages 33-36, Boulder, Colorado, Association for Computational Linguistics.
We focus on simple approach of curating small finetuning datasets that describe the labels for text classification tasks. Although this is beneficial when the task is specific, especially when the data is difficult to obtain, the data curation process is in- trinsically intuitive and relies on the practitioner's understanding of the labels and usage situation. Moreover, since a pretrained model is necessary for this approach, a few curated examples may miti- gate, but cannot detect or eliminate, potential biases of the pretrained model. If the labels of a certain classification task are dissimilar from the examples the model was trained on, and the model lacks the knowledge to differentiate among them, it may lead to unsatisfying performance even after finetuning a few examples of label descriptions.
Angelo Basile, Mare Tranco-Salvador and Paolo Rosso. 2022. Unsupervised ranking and aggregation of label descriptions for zero-shot classifiers. In Natural Lan- guage Processing and Information Systems- 27th International Conference on Applications of Natu- ral Language to Information Systems, NLDB 2022, Valencia, Spain, June 15-17, 2022, Proceedings, vol- ume 13286 of Lecture Notes in Computer Science, pages 119 196.1.6.666666 Springer.
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Duo Chai, Wei Wu, Qinghong Han, Fei Wu, and Jiwei Li. 2020. Description based text classification with reinforcement learning. International Conference on Machine Learning, pages 1371-1382. PMLR.
8 Ethics Statement
We use pretrained models for text classification, and curate data with the assistance of data sources such as Wikipedia and dictionary definitions. The large pretrained models are trained on a massive amount of data and have been shown to have issues with bias; however, this is common challenge when working with pretrained models and would benefit from advances made by the community on definitions
Ming- Wei Chang, Lev-Arie Ratinov, Dan Roth, and Vivek Srikumar. 2008. Importance of semantic rep- resentation: Dataless classification. In Proceedings of the Twenty-Thirl AAAI Conference on Artificial Intelligence, AAAI 2008, Chicago, Illinois, USA, July 13- 2008, pages
AAAI Press.
Zewei Chu, Karl Stratos, and Kevin Gimpel. 2021a. NATCAT: Weakly supervised text classification with naturally annotated resources. 3rd Conference on Automated Knowledge Base Construction.
this front. While both dictionary.com and Wikipedia are aimed at providing accurate and neutral information for a word/concept, they can be affected by the biases and limitations of their editors, especially for Wikipedia, which is an open- source encyclopedia. Our method is not reliant on specific dictionaries or encyclopedias; others could be used. We chose these resources for simplic- ity as they are highly accessible and widely used, Since our LABEL.DESC data is very small size, we manually examined the data as we selected for any potential biases or other issues. Finally, we use standard topic and sentiment datasets for evaluation, which are used in a great deal of prior work.
Zewei Chu, Karl Stratos, and Kevin Gimpel. 2021b. Unsupervised label refinement improves dataless text classification. In Findings of the Association for Computational Linguistics: ACI-IJCNLP 2021, pages
Online. Association for Computa- tional Linguistics.
Yann N. Dauphin, Gökhan Tür, Dilek Hakkani-Tinn and Larry P. Heck. 2014. Zero-shot learning and clus- tering for semantic utterance classification. In 2nd International Conference on Learning Representa- tions, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186. Minneapolis, Minnesota. Association for Computational Linguistics..
References
Libby Barak, Ido Dagan, and Eyal Shnarch. 2009. Text categorization from category name via lexical ref- erence. In Proceedings of Human Language Tech- nologies: The 2009 Annual Conference of the North American Chapter of the Association for Computa- tional Linguistics, Companion Volume: Short Pa-
Yu Fei, Yifan Hou, Zeming Chen, and Antoine Bosselut. 2023. Mitigating label biases for in-context learning. arXiv preprint 23Xiv:2355.11111

---

Yu Fei. Ping Nie. Zhao Meng, Roger Wattenhofer, and Mrinmaya Sachan. 2022. Beyond prompting: Making pre-trained language models better zero- shot learners by clustering representations. CoRR, abs/2210.16637..
Kamal Nigam, Andrew Kachites McCallum, Sebastian Thrun, and Tom Mitchell. 2000. Text classification from labeled and unlabeled documents using em. Ma- chine learning, 39(2):
OpenAl. 2022. Openai api [text-davinci-003]] https: //api. openai. wwt.c/oo/////o. /tcpmmllttiooss...
Ariel Gera, Alon Halfon, Eyal Shnarch, Yotam Per- litz, Liat Ein-Dor, and Noam Slonim. 2022. Zero- shot text classification with self-training. CoRR, 115422.0.17541.
Ethan Perez, Douwe Kicla, and Kyunghyun Cho. 2021. True few-shot learning with language models. In Ad- vances in Neural Information Processing Systems 34: Anmual Conference on Neural Information Process- ing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pages 11054-11070.
Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. arXiv preprint
Christoph H. Lampert, Hannes Nickisch, and Stefan Harmeling- 2009. Learning to detect unseen object classes by between-class attribute transfer. In 2009 IEEE Computer Society Conference Computer Vision and Pattern Recognition (CVPR 2009), 20-25
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018. Improving language under- standing by generative pre-training.
June 2009, Miami, Florida, USA, pages 951-958. IEEE Computer Society.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. Journal of Machine Learning Research, 21:1- 67.
Ken Lang. 1995. Newsweeder: Learning filter net- news. In Machine Learning, Proceedings of the Twelfth International Conference on Machine Learn- ing, Tahoe City, California, USA, July 9-12, 1995, pages 331-339. Morgan Kaufmann.
Timo Schick. Helmut Schmid, and Hinrich Schützee 2020. Automatically identifying words that can serve as labels for few-shot text classification. arXiv preprint arXiv:2010. 13641.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1 1907.11692.
Timo Schick and Hinrich Schütze. 2021. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Asso- ciation for Computational Linguistics: Main Volume. pages 255-2699, Online. Association for Computa- tional Linguistics.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. Fantastically ordered prompts and where to find them: Overcoming few- shot prompt order sensitivity.
Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 8086-800 8098, Dublin, Ireland. Association for Compu- tational L Linguistics.
Timo Schick and Hinrich Schütze. 2022. True few-shot learning with Prompts- A real-world perspective. Transactions of the Association for Computational Linguistics, 10:716-731.
Andrew L. Maas, Raymond F. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. pages 142-150, Portland, Oregon, USA. Association for Computational Lin- guistics.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013, Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 Conference on Empiri- cal Methods in Natural Language Processing, pages 1631-16422 Seattle, Washington, USA. Association for Computational Linguistics.
Yu Meng, Jiaxin Huang, Yu Zhang, and Jiawei Han. 2022. Generating training data with language models: Towards zero-shot language understanding. arXiv prepriat XxXv:2202.00553:
Yangqiu Song and Dan Roth. 2014. On dataless hi- erarchical text classification. In Proceedings of the TieentyEEihtth AAAI Conference Artificial Intel- ligence, July 27 -31, 2014, Québec City, Québec, Canada, pages 1579-1585. AAAI Press.
Yu Meng, Yunyi Zhang, Jiaxin Huang, Chenyan Xiong, Heng Ji. Chao Zhang, and Jiawei Han. 2020. Text classification using label names only: A language model self-training approach. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9006-9017. Online. Association for Computational Linguistics.
Chi Sun, Luyao Huang, and Xipeng Qiu. 2019. Uti- lizing bert for aspect-based sentiment analysis via constructing auxiliary sentence. arXiv preprint 188.0:1903.09588.

---

Mozes van de Kar. Mengzhou Xia, Danqi Chen, and Mikel Artetxe. 2022. Don't prompt, search! mining- based zero-shot learning with language models. In Proceedings of the 2022 Conference Empirical Methods Natural Language Processing, pages 7508-7520, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Xiaolong Wang. Yufei Ye, and Abhinav Gupta. 2018. Zero-shot recognition via semantic embeddings and knowledge graphs. In 2018 IEEE Conference Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 6857-6866. Computer Vision Foundation / IEEE Computer Society.
Qizhe Xie, Zihang Dai, Eduard Hovy. Thang Luong, and Quoc Le. 2020. Unsupervised data augmenta- tion for consistency training. Advances in Neural Information Processing Systems, 33:6256-66 6268.
Wenpeng Yin, Jamaal Hay, and Dan Roth. 2019. Bench- marking zero-shot text classification: Datasets, eval- uation and entailment approach. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP), pages 3913-3923, Hong Kong, China. Association for Computational Linguistics.
Jingqing Zhang, Piyawat Lertvittayakumjonn and Yike Guo. 2019. Integrating semantic knowledge tackle zero-shot text classification. In Proceedings of 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume (Long and Short Papers), pages 1031-1040, Minneapolis. Min- nesota. Association for Computational Linguistics.
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for classi- fication. Advances in neural information processing systems, 28:649-657.
Zihao Zhao, Eric Wallace, Shi Feng. Dan Klein, and Sameer Singh. 202 Calibrate before use: Improv- ing few-shot performance of language models. In In- ternational Conference Machine Learning, pages 12697-12706. PMLR.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021b. Calibrate before use: Improv- ing few-shot performance of language models. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Vir- tual Event, volume 139 of Proceedings of Machine Learning Research, pages 12697-12706. PMLR.

---

A Appendix
type
patterns
1 Question: What is the topic of this
A.1 Verbalizers
article? Answer: [MASK].
Q&A
2
Question: What the category of this
Dataset
Verbalizers
article? Answer: [MASK]]
3 Question: What the topic of this
20NG
talk.religion.miis misc religion, rec.autos + automobile, sci.med medicine, talk.politics.gume gun
article? Answer: [MASK]
4
Question: What the category this
article? Answer: [MASK]
AGNews
World - World, Sports Sports, Busi- ness + Business, Sei/Tech -> Tech
1
Category: [MASK].
PROMPT
2
Class: [MASK].
3
Topie: [MASK].
Yahon
Society & Culture + Society, Science & Mathematics - Science, Health + Health, Education & Reference - Education, Com- puters & Internet + Computer. Sports - Sports, Business & Finance + Business, Entertainment & Music - Entertainment, Family & Relationships -+ Relationship,
4
Theme: [MASK].
5
Category: [MASK]
6
Class: [MASK]
7
Topie: [MASK]
8 Theme: [MASK]
[MASK] News:
[MASK] NEWS: æ
Politics & Government ~> Politics
DBPedia
Company >> company, Educational institu- tion + schonl, Artist + artist, Athlete + sports, Office holder ++ politics, Mean of transportation + transportation, Building + building, Natural place + natural, Vil- lage + village, Animal + animal, Plant + plant, Album + album, Film + film, Written work + book
Table 11: Patterns for AGNews, where x refers to the
given text.
has the highest dev accuracy. The tuned patterns are listed in Table 14.
Yelp-5 SST-5
Very Negative + terrible, Negative ++ bad, Neutral - okay, Positive + good, Very
To our knowledge, this method works well when adapt to other datasets. However, also ob- serve that there are fluctuations in the dev accuracy curve for 20NG during training, and we select the training steps in the middle of the flatter part of curves rather than the peak point for robustness. We suggest changing training steps or increasing batch size if this method doesn't 't work well.
Positive + great
Yelp-2
SST-2
Negative awful, Positive + great
IMDB
Amz-2
Table 10: Verbalizers selected for each datasct.
The tuned pattern is not necessarily the best pat- tern after adapting to other datasets, sometimes little lower than the average results over all 14 patterns.
A.2 Patterns for MLM
A.2.1 Topic Classification
We use the patterns shown in Table 11 for AGNews and DBPedia, and replace "news/article" by "ques- tion" for Yahoo Question, which follows Schick and Schütze (2022)'s practice. We use "news- group" instead of "qucstioo""" " for 20NG.
A.4 Domain Transfer
All results on RoBERTa-baseelarge are shown in Figure 3.
A.2.2 Sentiment Classification
Our sentiment classification datasets (Yelp-2/5, SST-2/5, Amz-2, and IMDB) share the same pat- terns listed Table 12.
A.5 LABELDESC Data
The statistics of LABELDESC data are shown in Table 15. We use the same set of LEBELDESC data for AGNews and YahOoAG. Yelp-5 and SST- 5, Yclp-2 and SST-2, respectively. The data is listed in Table 16 Table 21. Each term/sentence that separated by "I" in tables is an independent LABEL DESC example during training. For brevity, we list all hand-crafted templates instcad listing all data for sentiment classification.
A.3 Hyperparameters and Best Pattern
We selected training batch size as 1 for experi- ments LESELDESC data. After fine-tuning on 20NG, the hyperparameters are selected shown in Table 13. With the selected hyperparameters, we further examine the dev accuracy on 20NG for all prompt patterns and select the tuned pattern that

---

type
patterns
pattern id
Question: What is the sentiment of
base
prompt
9
this text? Answer: [MASK].
LDT
Q&A
large
prompt
7
x Question: What is the writer's opinion
this text? Answer: [MASK].
MLMM
MISMATCHED
basc
qu
3
z Question: What is the sentiment of
large
qu
this text? Answer: [MASK]
RANDOM
basc
qu
3
x Question: What is the writer's opinion
large
prompt
6
this text? Answer: [MASK]
Table 14: Tuned pattern and pattern id for each model.
x Opinion:
[MASK].
PROMPT
2: Feeling: [MASK].
x Sentiment: [MASK].
I Summary: [MASK].
dataset
#label
LD
dev
test
z Opinion: [MASK]
20NG
4
24
3389
z Feeling: [MASK]
xr Sentiment: [MASK]
AGNews
4
24
2,000
7,600
x Summary: [MASK]
Yahoo
3,000
36,000
[MASK] Sentiment: x
Yahoo
60
60,000
10
[MASK] SENTIMENT: x
DBPedia
14
84
70,000
Table 12: Patterns for sentiment classification, where refers to the given text.
Yelp-S
2,500
50,000
5
SST-5
1,101
2,210
Yelp-2
2,000
38,000
SST-2
872
1,821
steps
2
100
Amz-2
400,000
LDT
base
5e-7
2160
IMDB
25,000
large
3e-7
1920
base
5e-5
2160
MIM
MISMATCHED
large
5e-6
3000
Table 15: Statistics of datasets we used, with "#" de- noting the number of labels, LD refers to LABELDESC
base
5e-5
2160
RANDOM
large
5e-6
3240
data.
base
lc-5
1920
classifier
large
le-6
2280
Label malk.
Type
Training Data
Table 13:
Hyperparameters (leariin rate, training steps) selected by tuning on 20NG with RoBERTa.
terms
religion Christian Buddhist lJewish
religion. Wiki. Religion usually delined social-cultural mise
designated behaviors and practices, morals, beliefs, worldviews, texts, sanctified places, prophecies, ethics, organizations, generally relates humunity super- natura, transcendental, and spiritual elements: however, there scholarly consensus what precisely stitutes religion.
A.6 Dataset Preprocessing
For 20NG, we remove headers, quotes, and footers. For AGNews, we concatenate the hcadlines and the text body of the news articles. For Yahoo dataset, concatenate the title, the question, and the top answer it. And for IMDB and Amazon Reviews Polarity datasets, we concatenate the title and the content.
diet.
of beliefs concerning the nature, and purpose the universe, especially when considered the creation of superhuman agency agencies, usulll involving devotional
and often containing moral code governing the conduct of human affairs.
lentts
automobile I truck car vehicle cur automobile)) used for transportation.
reciaulos
wheeled
diet.
passenger vehicle designed for operation ondinary roads and typically having four wheels and a gasoline or diesel intenal-centuuttion engine.
A.7 Label-wise Metrics
medicine hospital symptom cure
sei melmme
Wiki, Medicine the science and practice curing a paticnt,
We list label-wise precision, recall, and Fl scores for part of our datasets in Table 22 - 29.
manaping the diagnesis, prognosis, prevention, treatment, pulliation
injury disease. promoting
health.
diet. any substance
in treating discase
illness; medicament; remedy.
ullk
lenus
gun lirearm I weagon handgun gun ranged wespon dosigned
poliries,
shooting
guns
(gun barrel) to luncc projectiles.
diet.
artchmenrs, consisting of metal tube, with mechanical projectiles shor by the of an explosive: a piece of ordnance.
Table 16: LEBELDESC data for 20NG. "Wiki." and "diet." refers to the data source, i.e., Wikipedia or dictio- nary definition.

---

Label
Type
Training Data
terms
world country international politics
World
Wiki.
its most general seuse, the term "world" refers the totality of enrities, to whole of reality or everything that is.
AGNews test set (base)
AGNews set (large)
900
1000 aecunucy (36)
diet.
humankind;
race, humanity
80
sport sports raacng buseball
85
Sports
Wiki.
Sport pertains to any form of competiivve physical activily game that aims ability and skills while providing enjoyment to partiel- punts and, un athletic activity requiring skill or physical prowess and often of comperitive nature, as meing. basehall, tonnis, golf, bowling. wrestling, boxing, hunting, fishing. etc.
maintain or improve phy sical
80
75
entertainment spectators.
diet.
100
500
10
100
500
Yathoo test set (base)
Yahoown test set (large)
8G
terms Wiki.
business finance money trade
Test accunnyy (8)
Business
Rusiness is the acriviry of makinng one't 's living or making money by producing or buying and selling products (such as goods and services).
75
70
diet.
the purchase profit.
goods attempt
85
60
letra Wiki.
technology sccience computer biology
Soi/Tech
Technology the continually developing result cumulated knowledge and application in all techniques, skills, methods, and processes used in industrial produc- tion and scientific research.
100
500
10
500
Yelp test set (base)
Yelp- 5 test set (large)
diet.
the brunch of knewledge that deals with the creation and use of technical means and their interelatioon society, and the environment, drawing upon such subjects industrial engineering, applied science, and pure science.
securnccyy (5)
life,
60
65
50
45
Table 17: Yahoo
LABELDESC data for AGNews (and
10
1000
500
10
100
500
SST-5 test sel (base)
SST-5 test sel (large)
65
1est accurncy (8)
50
45
Label
Type
Training Data
40
Very
temms
awful terrible borrendous horrible I dreadiul was f. Afn) experience. Just 1.1 Overall, was i.
95
Negative seat.
100
500
10
had implessant unsatisffyngg I lonsy subbaar It was t. AAaa) experience. Iust 6. Overall, was t.
Negative
Yelp-2 set (base)
Yelp-2 test set (large)
seat.
Rest sccuracy (26)
Neutral
temms
okay mediocre docest I lveeaage I alrightt was t. A(n) experience. Just Overall, was
9.5
good nice fime pleasant neal
Positive
sent.
It was 1, A(n) ( experience, Just t. Overall, was
80
Very
terms
great lamazing excellent fantastic I outstanding f. Afa) expetionce. Just Overall, was
Positive
seat.
10
200
500
10
100
5000
Table 18: LABELDESC data for Yelp-5 and SST-5. "Sent." and "I" refer to hand-crafted sentence templates and terms, respectively.
SS 1-2 test set (base)
T-2 test set (large)
8.5
lest aocunccy (36)
80
85
80
75
100
10
Label
Type
Training Data
in-domain
out-of-domain
LDT
Negative
terms awfal temrill horreadoos I horrible I dreadful had unpleasant unsatisfying lousy subpar
soot. It was f. Atn) ( experience, Just &, Overall, was (.
Figure 3: Domain transfer results, where X-axis depicts the number of training examples per label. "base/large" in parenthesis denotes RoBERTa-base/lann
Positive
terms
good nice fine pleasant neat great amazing excel.
lent fantastic buistanding
S00t. It was f, Atr) experience, Just &. Overall, was t.
Table 19: LABEI.DSC data for Yelp-2, SST-2, Amz-2 and IMDB.

---

Label
Type
Training Duta
Soclety & Culture
terms
sociery calltare
Wiki.
society group individuals involved in persistent social interaction, large social group sharing the spatial or social territory. typically subject the same political authority and dominant cultural expectations. CCulture an umbrella term which encompasses the sncial hehavior, instituzions, and norms found in human sacieties, as well as knowlegee. beliefs, arts, laws, customs, capabilisies hubits of individuals these group.. an organized group of persons nssociated together for religious, benevolent, cultural, scientifie, political, patriotiic, or other purposes. the behaviors and beliefs churacteristic particular group of people, as or age group (usually used in combination)
dict.
ethnic, professional.
Science
lerma
science mathenaticss
Wiki,
Science systematic endeavor builds and organizes knowledge the form of testable explatations and predictions about the universe. Mattematiics is an area of knowledge that includes such tapics as numbers, formulas and related structures, shapes and the spaces in which they contained, and quantilies and their changes.
Mathematics
dicr.
beanch knowledge or study dealing with hody of facts tuthe systematically arranged and showing operation of general laws the systematic treatment of magnitude, relationships between figures and forns, and relations between quantities expressed symbolically.
terms
health litness medical diet
Health
Wiki.
Health, acconding to the World Health Organization, is "a state of complete physical, mental and social well-heing and not merely the absence disease infirmity"
dict.
the gencral condition of the body or mind with reference to soundness and vigor
Education
terms
education I reference
Wiki,
Education a purposefull activity directed achieving vettain aims, such as tramsantting knowledge fostering skills and character traits. I Reference is relationship berween objects in which one object designatss or ncss as a mecas by which connect or link to, another object.
Reference
dict.
the act or process of importing or acquiring general ledge, developing the powers reasoning and judgmment, and generally of preparing omesell or others intellectually for mature life. l book or other source useful facts infomution, such as an encyclopedia, dictienary, etc.
Computers
terms
computer internet
& Internet
Wiki,
computer digital clectronic machine that can be programmed to out sequences of arithmetic logicall operations (computation) antomarically, The Internet intemet) the glohal system interconnected competer networks that uses the Internet protocol suite (TCP/I)) communicate beiween networks and devizes. programmable electronic device designed accept data, perform preseribed mathematical at high speed, and display the results these operations. Mainframes, desktop and laptop computers, tablets, and smanphones some of the different types computers. Ussally the internet (except when used before noun). vasi computer network linking smaller computer netwrrk worldwide. The internet includes commercial. educational, governmental, and other networks, all of which use the same set of communications protocols
diet,
logical operations
terms Wiki,
sport I sports racing basehall
Sports
Sport pertains to any form of competitive physical activity or game that aims to use, maintain or improve physical
ahility and seills while providing enjoyment to parricipants and, in some cases, enteraiimment to spectators. an athletic activity requiring skill or physical prowess and often of competitive nature, as racing. buseball, tennis, golf, bow wing wresting, boxing, bunting, fishing, etc.
dict.
Business
terms
husiness finance
Wiki,
Business the activity making one's living ur making money by producing or buying and selling products (sach as goods and services). I Finance is the study and discipline of money, currency and capical assets.
Finunce
dict.
the purchase sale goods an attempe make a profit, the management revenues; conduct transaction
of money matters generally, especially those affecting the public, as in the fields of banking and investment.
Entertaimment
terma
entertainmentt music
Wiki.
of audionse or gives pleasure and dalight. Music is generally defined as the art arranging sound to create some combination of form, harmony, melody, rhythm otherwise expressive content.
Fntertain inment form activity that holds the attention and
Music
dicr.
the act entertaining; agrecable occupation
mind; diversion; ammsement an art sound in time that expresses ideas and emotions significant forms through the elements of rhythm, melady, harmony, and color.
Family
family relationship
Wiki,
Hamily is a group of people related either by consanguinity (by recognized birth) or allinity (by marringe or other relationship). I The concept of interpersonal relationship involves social associations, connections, or affiliations between two or more people.
Relarionships
dict.
basic social unit consisting of parents and their childron, considered as a group, whether dwelling togather or not, social unit consisting of one more adalts together with the children they for. an emotional other connection between people
Politics
terms
politics I government
Wiki,
Politics the set of activities that associated with making decisions groups, other forms power relations among individuals, such as the distribution of resources or status. A government is the system or group of people governing organized community. generally state. the science art political government. the political direction control exercised over the actions members, citzens, or inhobitants of communities, sucieties, and stales; direction of the affairs of a state, community, etc.; policical administration
Government
dier.
Table 20: LABELDESC data for Yahoo Answers.

---

Label
Type
Training Data
lerms
compuny lizm corporatioo business
Company
Wiki.
company, ahbreviated
is logal carity reprosenting association people, whather natural, legal
mixture of both, with a specific objective.
diet.
number of persons united incorporated for joint action, especially for business
echcationnal institurion university college I school
Educational
Wiki.
An educational institution is place where people of differenti ages gain an education, including preschouls, chilldcure, primary elementary schools, secondary high schools, and universities.
Institution
dict.
an institution for instraction in a particular skill or field.
terms
anist writer actor singer
Artist
Wiki,
An urtist person engaged in an activity related creating practicing the demomstrtiig
dier.
person who produces works any of the arts that are primarily subjoct to aesthetic criteria.
terms
shlete I sports foothaller weightlifter
Atblett
Wiki,
An athlete (also sportsman sportsooman) penson who competes in sports that involve physical
strength, speed, endurance.
dict..
person trained or gifted in exercises ur contests involving physical agility. staminz, or strength: participant in sport,
exercise, or game requiring physical skill.
Office Holder
terms
oolcc-hllder politics I mayor president
Wiki.
penson who's is heen appointed to position by company or organisation hot doesn' 't have a contrct receive regular holder.
payment may be
dict.
person filling governmental position: public official.
Mean
terms
mean of transpoctation car I hus trann
of Transportaionn
Wiki,
Transpont Brilish English), ot transpcttation (in American English), intentional movennent humans, animals,
and goods from one location to another.
dict.
meuns of transportng conveying, truck or bus.
building lpprttment skyscruper lower
Building
Wiki.
building or edifice, an enclosed stracture with roof and walls standing more or less permanently in one place, such house factory (although there's is also portable huildings).
dict.
relatively permanent enclosed construction over plot of land, having roof and usually windows and often more than level, used for
wide variety of activities, living, entertaining, manufacturing.
natural place! forest mountain river
Natural
Wiki,
The matural envir envirocment or natural world encompusses all living and zom com-liingg things occurring maturally, meaning in this ease not artificial.
Place
dict.
existigg in formed by nature (opposed to anificial)
terms
village town couniryside rural
Village
Wiki.
A village is clistered homan settlement or community, larger than hamler but smaller than town (although the word used descrihe both hamlets and smaller towns), with popalation typically ranging from few hundred to few thousand.
dict.
small community group houses rural area. larger than a hamket and usually smaller than town, cometimes (as in parts of the U.S.) incorporated as municipality.
terms
animal insect bird fish
Animal
Animals are multicellular, cukaryotic organisms in biological kingdom Animalia,
diicr.
any member of the kingdom Animalia, comprising multicellular organisms that have a well defined shape and usually voluntarily, actively auquire food and digest internally, and have semsory and nervous systems that allow them to respond rapidly to stimali: some classification sehemes also include procozon and certain other singlo-cellid cukaryotes that have motility and animallike nutritional modes.
limited growth.
plant flower! tree grass
Plant
Wiki, diicr.
Plants are predoninaaaaly phokosymihetic eukaryotes, forming the kingdom Plantae.
Botany, any member of the kingdom Plantac, comprising multicellular organisms that rypically produce their own food from inoeganic matter by the pracess of photosymthesis cdllulose, including vascular plants mosses, liverwerts, and bornworts: some classificution schemes may include fungi, algne, hacteria, and certain single-celled eukaryotes that have plantlike qualities, rigid cell walls the of phocosynibesis.
more or less rigid walls containing
terms Wiki,
album I soundrack mixtape I CD
Allbam
An albom collectinn of audio recordings issued on compact disc (CD), vinyl. audio tape, another medium such as digital distribution.
dist..
record
records contnining several musical selections, complete play opera,
Ilm moviel comedy drama
Film
Wiki.
film also called a movie, motion picture, moving picture, picture, photoplay or (slang) flick is work of visual art
that simulates experiences and otherwise communicntes ideas, stories, perceptions, foolings, beauty, aimosphere
through the
moving images.
dict.
sequence of consecutive still images recorded in a series to be viewed on screen in such rupid succession as to give the illusion of natural movement, motion picture.
Written
terms
writen work novel newspaper hook
Wiki,
A book a medium for racording information in the form of writing or images, rypically composed of many pages
Work
(made papyrus. parchment, vellum. or paper) bound logether protected by cover. handwitten or printed work of fiction or confiction, usually on shoets of paper fastened or bound together within
distt
Table 21: LEBELDESC data for DBPedia.

---

precision( (%)
recall (%)
FI(%)
dataset
RoBERTa
label
zero-shot
LDTT
zero-shot
1.TT
zero-shot
L.TT
World
58.7 122
80.2173
29.1 1218
62.0 18.2
33,7
68.0=1200
Business
60.6#*.1
71.0+63
66.9-14m
77.0=46
63.0=97
73.7=19
base
Sports
1477
94.1
92.5
94,4
80.1
94.1=33
Sci/Tech
65.3+158
69.9+92
62.4=148
76.4-60
60.6=x1
72.4-43
AGNews
World
81,6=100
78.8
53.1
84.1
61.5a15.
81.0=13
Business
53.1+1377
67.4+99
84,6-92
86.4-5a
63.6-71
74.9-47
large
Sports
86,8453
95,6+12
90.4
90.2-7a
88.2=39
92.7=45
Sci/Tech
75.0=123
87.0+48
44.1=119
57.0-13.2
55.0=114
67.8=93
World
44,8
64.4+44
34.9
62.9
35.8-138
62.7=53
Business
45.2=43
53.1
48.7=66
52.9
46.5=32
52.3=27
hase
Sports
49,4
86.7258
83.7
78.7=76
57.1
82.1=39
Sci/Tech
72.9=14.1
70.0=s5
45.3
71.2=74
52.7=127
69.9
YahonaG
World
63.4+9.
77.2+56
25.6=208
53.8-120
32.6=156
62.5-1a
Business
33.4200
47.3286
70.2=89
63.7=83
44.1=41
53.1=30
large
Sports
61.22157
88.3143
85.9
82.9_57
69.4
85.3=25
Sci/Tech
72.9
72.4
50.9
78.7_93
59.5275
74.5_38
Very Negative
67.1 1228
68.6144
19.6
14.
8.3
46,4=124
Negative
34,5
41.5+23
75,2=167
55.7=109
46,2=37
46,9-28
base
Neutral
24.2
53.0129
2.5
14.1=5.1
4.3
21.7-66
Positive
34.4#61
29.2+23
50.8=23.2
16.3-50
38.4=10.
20.5=43
Very Positive
59.6=118
42.0=21
56.2=25.0
94.0=1a
52.42118
58.0=18
Yelp-5
Very Negative
78.61202
82.9129
12.8
34.3
16.2.205
47.2-11.9
Negative
39.4#s2
44.7=26
55.0=219
75.8
43.2-123
56.0=1.1
large
Neutral
39,2+22.
64.2+2*
4.4+77
20.7
7.0+105
30,2-99
Positive
31.5+56
38.5+19
84.8=99
36.8
45.4=s6
36.8=17
Very Positive
72.1
56.2+53
36.7=218
88.8
44.2=22.
68.2=24
Very Negative
13.7215.
26.9231
4.448.
24.2-74
5.7186
24.7=43
Negative
45.0=17
53.2+16
60.3=29.8
58.0=74
46.4286
55,2=30
hase
Neutral
8.92115
26.0233
0.7418
11.7=76
1.2139
14.9_66
Positive
33,0a54
39.5
57.5
30.1
18.1
33.3
Very Positive
45.9
42.5133
24.3
73.6=70
22.9
53.6-15
SST-5
Very Negative
21.1
35.9266
13.62216
21.2-74
11,2=140
25.8=57
Negative
51.7147
54.5114
36.72268
73.7-65
37.62a12
62.5_20
large
Neutral
7.54130
32.3+68
0.7+17
6.7+38
1.2+29
10.8=55
Positive
31.3161
46.2122
91.5299
52.6-127
46.0as8
48.2.119
Very Positive
66.8+25.6
53,4+664
8.3+
67.2-13.
12.1=158
58.0=40
Negative
91.8
97.01a9
27.6
79. 79.1-52
38.7
87.0=32
base
Positive
58.8+7.3
82.5+38
99.7=03
97.5-u9
73.7=57
89.3-19
Yelp-2
Negative Positive
99. 2904
97.61:09
41.4
91.6-41
51.5
94.4=21
large
65.5+135
92,2+13
99.7=a3
97.7
78.3+96
94.8-15
Negative Positive
85,5
81.8+44
28.9
89, 19.1
37.7=29.
85,2=19
base
58,8+s4
88.3+311
96.5=33
79.9
72.6=54
83.7=29
SST-2
Negative Positive
83.2+15.3
94,5+13
28.8-2299
88.0-54 94.7
36.5 74.0m79
91.0-26
large
19.9
89.0
98.8
91.7=11
Table 22: Precision, recall, and score for zero-shot LABELDEECCTE DESCTRAINING (for brevity, LDT header).

---

precision(%) (%)
recall (6)
(1())
dataset
RoBERTa
label
zero-shot
LDT
zero-shot
LDT
zero-shot
LDT
Society Culture
27.2191
39.0261
2.7417
8.8112
4,5
14. 1442
Science & Mathematics
40.0
60.4
68.5
58.8
47. 17.4
59.3424
Health
52.1+69
59.1+60
74.9=120
77.6+42
60.4+47
66.6+34
Education & Referrnce
39.8+-122
46.5
22.0=125
36.8+49
24.6+104
40.6+25
Computers & Internet
80.1+922
69.1=3*
19,2+115
78.2+50
29.6+143
73.1+19
base
Sports
70.7=2116
81.9268
65.92218
81.7=23
61.82115
81.5+1a
Business & Finance
41.4+229
54.4=30
34.9=89
44.7+a2
36.1
48.9+1.7
Entertainment & Music
49.4
62.5=56
39.52203
58.0244
38. 38.72135
59.812.1
Family & Relationships
70.0199
47,2
30.21146
81.2.98
40 20121142
59.2125
Politics & Govemment
46.5
64.5271
27.4
62.2174
40.62178
62.6124
Yahoo
Society Culture
35.3294
46.1=70
2.2+40
9.6148
3.7=57
15.5
Science & Mathematics
18.6
65.7
67.1211
66.2214
54.3166
65.6122.
Health
54, 10.7
65.5
82,5
81.2+34
63.7.33
72.3+155
Education & Referrnce
33.2:1149
38.3
34.1
46.9169
30.2.1955
41.144
Computers & Internet
81.8
80.2=38
36.2+16.9
75.1+53
47.3+1126
77.3+21
large
Sports
75.8+11..
87.8=55
74.9=17.5
83.3+39
72.1#110
85,3+23
Business & Finance
37.2++9.4
56.2#52
46.5+144
43.9+57
38.7+60
48.8+24
Entertainment & Music
52.7=82
64.5=49
38.3=2200
58.9+67
39.7=171
61,2+36
Family & Relationships
71.2284
47.5=s0
47.5
85.0288
50.84258
60).419
Politics & Govermment 57.5
71.8=50
48.0=30.
57.5+45
46. 116.1
63.5+14
Table 23; Precision, recall, and Fl score for zero-shot and LABELDDESCTRAIING
In-domain
Oft-of-domain
AGNews
zero-shot
LDT
10
100
500
10
100
500
World
58.7.128
84.7126
88.5 1445
93.0=26
77 17.4
81.2155
79.8168
80.2
Business
60.6#*a
80.9+4.
83.0438
86.9
75.2+6.1
72.8
73.1
71.0=63
Prec.-b
Sports
72.9+147
94.7+1.3
95.7+11
96.6=n6
91.6+32
92.1+3a
94.4=a7
94.1=15
Sci/Tech
65.3+158
79.6*66
85.8+45
86.6#26
72.6+9.
78.6470
83.2+49
69.9=92
World
1271
85.7+36
87.1+30
88.7=34
78.7+104
78.8+5.1
81.0=s2
62.0=182
Business
66.92140
77.4274
83.6175
86.4227
63.32145
76.22114
81.7256
77.0246
Rec.-b
Sports
92.5
98. +os
96.9+24
98.2=05
97.422
98.420s
97.9=0a
94.4262
Sci/Tech
62.4
78.0258
84.2148
89.2233
72.5199
67.9110.
66.8281
76.4260
World
33.7=212
85.1+12
87.6+1.
90.7-1.
77.0144
79.7122
80.0=19
68.0=120
Business
63.0297
78.8137
82.9134
86.6228
67.3185
73.3148
76.9216
73.7239
Fl-b
Sports
80. 1287
96.41a0
96.241.3
97.4
94.4113
95.1
16.s
Sci/Tech
60.6+8.1
78,4+35
84.8+11
87.8+09
71.4437
71,9+45
73.6+36
72.4+43
World
81.6#100
84.7+23
89.0428
93.6-28
77.4-667
83,9442
81.4#62
78,8=63
Business
53.1
83.8+32
83.7+30
88.0=24
79.5+3.
74.0+5.
72.4=53
67.4=99
Prec.-I
Sports
86.8+53
95.2+111
96.3
96.9
93.1+33
93,0+23
94.6*a9
95.6=11
Sci/Tech
75.0=123
82.4+57
87.8+24
86.8=25
80.6+48
82.6157
87.1
87.0=48
World
53.1221a
89.0221
89.0213
89.2=36
86.7147
82.9114
82.4250
84.1
Business
84.6292
78.6168
85.8+34
86.8
70.4-99
79.3
83.6243
86.4
Rec.-l
Sports
90.428a
97.8109
97.911.
98.5200
98.5100
98.5107
97.7218
90.2278
Sci/Tech
44.1
80.2
83.6144
90.2a30
73.0.9.1
70.4199
67.5193
57. 57.0211.
World
61,52151
86.7108
89.010s
91.3212
81.413.1
83.2114
81.5
81.0243
Business
63.6
80.9
84,6110
87.3
74.04447
76.1
77.3
74.9
Sports
88. 2:19
96.5
97.1
97.7
95.7
95.7
92.7
Sci/Tech
55.0=114
81.0+2s
85.5
88.4
76.0
75.245.
75.5
67.8
Table 24: Precision, recall, and Fl for AGNews, where 'b' refers RoBERTa-base, 'I' refers RoBERTa-large.

---

In-domain
Out-of-domaa
Yahooa
zero-shot
LDT
10
100
500
10
100
500
World
44.8169
80.4
82.8119
85,4
76.0166
84.3
85.5143
64.4144
Business
45.2
48.4
53,3
56.2=52
63. 1315
58.1
50.2
53.1
Prec.-b
Sports
49.4+21s
83,8+67
86.6+6.1
91.2+25
81.9+138
89.6+53
88.9+s*
86.7+58
Sci/Tech
42.9
82.4+50
88.0+34
88.9#20
65.1477
62.3
60.0#47
70.0
World
34.9+253
67.4=102
75.9+38
77.7=54
62.5+162
50.1
37.6=135
62.9+126
Business
48.7268
61.7299
63.6198
68.9257
34.22167
46.1
48.3276
52.9162
Rec.-b
Sports
83.7+173
85.7=57
91.0+22
91.1=11
85.7+36
82.4+64
80.8=36
78.7+76
Sci/Tech
45.3
80.2256
81.7143
85.7
84.015.1
92.8114
94.2222
71.215.1
World
35.8+138
72.5449
79.0+25
81.2=17
66.5
61.4294
50.5
62.7+5a
Business
46.5132
53.3132
56.7133
61.4211
40.61150
49.8173
48.2222
52.3117
F1-b
Sports
57.1
84.4.22
88 28.6118
91.1
82.6 62.6
85.5
84.4
82.1
Sci/Tech
52.71125
81,0219
84.5
87.2.08
72.9133
74.1
73.0143
69.9130
World
63.4s9a
81.1
84.2+33
85.8
83.6460
90.0#26
89.3
77.2
Business
33.4+am
50.8+43
55.1+6.4
57.6-53
68.3+116
60.7+64
54.8+15
47.3+36
Prec.l
Sports
61.2+157
87.0#56
88.6+35
93.1+22
81.5+11
86.3
89.2+76
88.3++43
Sci/Tech
72.9+8.6
86.2#27
89.5+13
90.1=24
58.6+87
$2.6#3.9
56.7+17
72.4+s.1
World
25.6+208
69.4+17.
77.3
79.7=56
51.1+182
31.1+10.
33.4
53.8+120
Business
70.2
68.3=50
66.9+74
70.1
28.8+163
38.4+81
44.6296
63.7+43
Rec.-l
Sports
100
88.8 8246
92.411a
91.7-24
87.9148
86.4126
83.8245
82.9157
Sci/Tech
50.9289
81.0239
83.0144
86.1
90.2144
95.621.
95.6233
78.7
World
32.61156
74.5236
80.4128
82,4
60.81127
45.22113
46.921311
62.5178
Business
44.1
57.9
59.7
62,8
36.6115.1
46.2161
47.9
53.1
Sports
69.4166
87.6123
90.3124
92.3
83.8152
86.2122
86.0233
85.3
Sci/Tech
59.5
83.4+14
86.0
87.9
70.44449
67.8+33
70.8
74.5
Table 25; Precision, recall, and Fl for Yahoowa, where 'b' refers RoBERTa-base. 'I' refers RoBERTa-large.

---

In-domain
Out-of-doman
Yelp-5
zero-shot
LDT
10
100
500
10
100
500
Very Negative
67.1
65.5
71.1124
72.0120
51.5158
55.1
55.3147
6.44
Negative
34.5+44
44.6=29
51.6+16
$7.8+15
34.5
39.7=37
42.7432
41.5+23
Prec.-b
Neutral
34.21207
47.9
$2.7.39
53.012a
40.3
45.5a43
45.013.
53.0a29
Positive
34.4+6.1
44.7==2
49.7+22
54.1+12
36,4467
39.9=sa
39.243.
29.2+22
Very Positive
59.62118
61.7=35
70.323a
70.3
50.0
53.6=53
54.1
42.0=a.1
Very Negative
5,3+06
58.7=100
71.4+43
77.8+28
62.5+170
76,0=97
80.6+74
37.8-145
Negative
75.24167
18.5=108
55.3247
47.9
19.1
24.6=129
22.5
55.7=109
Rec.-b
Neutral
2.5
10.4
46.82102
59.7144
39.21184
35.0=111
35.62111
14.1
Positive
50.88233
44.0=99
53.8274
50.1
15.62163
24.42114
25.1
16.3250
Very Positive
56.21250
70.2287
64.41115
72.212.1
86.6174
83.7267
83.3166
94.0.18
Very Negative
8.3
60.8
71.1+1a
74.7
54.5
63.1+19
65.1
46.4+122
Negative
46.2133
46.0 0265
53.2116
$2.1
20.41147
28.52106
18.1
46.9
F1-b
Neutral
4.3+55
42.9=57
48.6+43
56,0+10
37.1+113
38.2+71
38.6+75
21.7+61
Positive
38.4
43.7=51
51.3+23
51.9+14
17.4+142
28.2=108
28.6=114
20.5
Very Positive
52.4+11a
65.2+25
66.8+29
71.2+0.
62.7+14
64.9=25
65.2+16
58.0+18
Very Negutive
78.62202
71.0a32
73.2231
75.8
57.3
59.8=155
61.7+5a
82.9
Negative
39.4+53
53.0=19
54.9+15
60.9
42.9+68
45,2=112
50.3+29
44.7+26
Prec.-l
Neutral
39.24228
54.8230
57.7-27
58.1
48.0259
46.72118
52.1426
64.2=228
Positive
31.5156
53.2122
54.4120
S7.2109
36.3193
44.0269
46.0133
38.5139
Very Positive
72.1189
70.7
72.7131
73.1
52.6134
18.1
60..7441
56.225a
Very Negative
12.81214
12.
76.61660
78.5123
61.21240
73.2
83.5166
34.3
Negative
55.0+258
47.0=68
56.8+6.2
53.1
19.0
26.5=149
31.24996
75.8
Rec.-l
Neutral
4,4
60.8
53.8169
62.613.1
40.01145
42.0
40.218.
20.7 20.7.18.
Positive
84.8499
49.2=70
54.6+57
56,4+2.1
22.3+
41.1
37.4+123
36.8+134
Very Positive
36.7
71.9
69.7
74.5
88.3+10.2
76.6=218
85.6+5.1
88.8
Very Negative
16.2+2us
71.2+23
74.6+11
77.1
55.7+11.5
64.5+153
70,5+1.7
47.2+1199
Negative
43.2+123
49.5=34
55.6426
56.6+26
30.1
31.1=136
37.7+76
56.0=1.1
F1-1
Neutral
7,0=105
57.2+24
55.3+30
60.2+a9
42.0+96
42,4=123
44.7+5.2
30.2+99
Positive
45.4256
50.8231
54.342.
56.8
22.5
39.4293
40.1
36.8277
Very Positive
44.2.222
70.1
70.8125
73.6114
64.8140
64.62159
70.8
68.2.22
Table 26: Precision, recall, and F1 for Yelp-5, where "b" refers to RoBERTa-base, 'I' refers RoBERTa-
In-domain 100
df 100
Yelp-2
zero-shot
LDT
10
500
Negative Positive
91.8+364 58.8+7.3
92.6#23
94.1=19
95.3+1.
94,8+20
94.3+22
92,8=25
97.0+09
Prec.-b
93.5444
95.3+17
95.1+27
89.6+32
88.9437
92.5-36
82.5438
Negative Positive
27.6:317
93.3+5.
95.3219
94.9
88.8
87.9
92.2=46
79.1
Rec.-b
99.7
92.4
94.0=22
95.3122
95.1
94.5129
92.7
97.5109
Negative 38.7+379
92.8+20
94.7=06
95.1
91.6219
90.9
92.4-18
87.0432
F1-h
Positive
73.7157
92.8
94.6227
95.1
92.2114
91.5114
92.5
89,3119
Negative 99.5404
95.7218
95.7217
97.1
96.6111
95.54119
95.8211
97.6209
Prec.-1
Positive
65.5
95.9122
97.3
97.2.06
95.0111
94.5
95.3
92.2433
Negative 41.42313
95.7433
97.4210
97.2100
94,9
94.3133
95.2222
91.6114
Rec.-l
Positive
99,7103
95.7.11
95.6211
97.1
96.7111
95.5122
95.7
97.7.09
Negative Positive
$1.5+333
95.7+11
96.5+aa
97.1+002
95.7+a7
94.8+
95.4=0a
94,44+2
F1-1
79.
95.7
96.4
97.1
95.8
94.9
95.5
94,8
Table 27: Precision, recall, and Yelp-2, where refers to RoBERTa-base,
refers RoBERTa-large.

---

In-domain
Out-of-domain
SST-5
zero-shot
LDT
10
100
500
10
100
500
Very Negative
13.7
34.9
43.4
44.7139
28.5
31.8
33.0122
26.9
Negative
45.0+
52.2+26
55.6+19
59,0+19
53.0+18
53.5=21
58.6+17
53,2+16
Prec.-b
Neutral
8.9
25.8
32.2.19
34.8
25,9
27.7
26.011s
26.0as3
Positive
33.0+54
44.5
49.1+22
51.8+27
41.6+47
42.8=44
43.5+55
39.5+24
Very Positive
45.9
54.2263
58.1247
57.9432
47.7
49.9=5a
49.5+24
42.5=a3
Very Negative
4.4+81
37.1+108
53.2+110
$7.6+a8
48.9+223
36.0=94
64.0+66
24.2+74
Negative
60.31220
47.4=9.
44.9=118
42.9470
45.7
59.6=105
38.5
58.0=74
Rec.-b
Neutral
0.7=118
24.2
32.8195
34.5
13.21127
23.1
37.7158
11.7276
Positive
57.54275
46.6295
51.4189
53.2484
30.7
32.5=180
14.8250
30.1294
Very Positive
24.31272
56.0
57.5194
66.9177
65.61139
52.2214
62.1170
73.6220
Very Negative
5,7=x6
35,0=s1
46.6433
49.7
33.2414
32.8=3.1
43.2+1.1
24.7+44
Negative
46.418.6
49.11:48
48.7.16
49.3
45.1
55.8244
46.0144
55,2130
F1-b
Neutral
1.2+29
23.5+82
31.5+46
34,0+4m
13,9498
21.4+11
30.6+24
14.9+66
Positive
38.1+114
44.8=46
49.8+40
52.0+4m
31.74152
34.2=130
21.7+57
33.3-68
Very Positive
22.94172
53.9+34
57.1+4.
61.7420
53.8+15
49.4=55
54.8+22
53.6+15
Very Negutive
21.1
35.2
41.1
45.7+30
34.1
33.0=37
33.2
35.9=60
Negative
51.7+47
52.0=34
55.0+130
59,6+18
53.5+41
53.2=34
58.1+23
54.5+14
Prec.-l
Neutral
7.5=130
26.6227
33.4188
38.6225
33.3+10
29.1
31.6228
32.3266
Positive
31.316.1
46.6272
19.1
55.81199
48,1
49.0246
51.114.
46.2126
Very Positive
66.81256
53.2258
56.62138
61.8131
57.1
52.9244
50.1146
53.4264
Very Negative
13.61236
39.1
54.51167
58.1
56.71123
58.32119
69 19.6118
21.2174
Negative
36,7+263
43.1
37.4+152
45.9
48.1
54.2=107
41.4+
73,7=65
Rec.-l
Neutral
0,7
25.
33.01154
38.2176
18.3
16.6
25.8133.
6.7
Positive
91.5499
43.2+108
58.1+147
55.2+17
43.0+13.
30.8=97
28.8+au
52.6+127
Very Positive
8.32120
65.4
58.42192
72.2
59.8
63.5=102
68.0=11.
67.2=133
Very Negative
11.2+149
36,3+s5
45.7+109
50,9+17
39.7+19
41,2+26
44.6+26
25.8+57
Negative
37.6#212
45.9=90
43.12139
51.5+47
47.7
52.9=46
47.7+53
62.5=20
F1-1
Neutral
1,2+29
24.8=57
31.4998
37.9+33
21.3+17
19.0=66
28.0+34
10.8=55
Positive
46.0255
43.3286
51.5455
55.1
44.0262
37.0=72
36.1269
48.2244
Very Positive
12.1
57.7
55.91146
66.3117
56.5158
57.0
56.812.7
58.0.40
Table 28:
Precision, recall, and F1 for SST-5, where "b' refers RoBERTa-base 'I' refers to RoBERTa-large.
In-domain 100
Out-of-domain
SST-2
zero-shot
LDT
500
Negative Positive
59.6=118
91.1+3a
93.7423
93.3a26
74.1
79.9+8.
83.6#59
81.8+40
Prec.-h
58.8=84
88.0+4a
88.6+44
91.9430
95.3-40
94.1+33
91.9452
88.3+31
Negative
28.9=25.a
87.2+35
87.546.
91.7237
96.1244
94.8
92.2
89.1
Rec.-b
Positive
96.5236
91.2116
93.9
93.3230
14.1
14.41146
80.8192
79.9259
Negative
37.7=200
88.9+13
90.3430
92,4
83.2=24
86.3433
87.2+17
85.2
F1-b
Positive
72.6254
89.41115
91.0118
92.5
15.1
81.91104
85.4142
83.7229
Negative
83.22383
94.5433
94.8122
95.1
88.3239
84.9462
88.7441
94.5223
Prec.-1
Positive
59.9-11.
92.1132
91,9140
93,9121
94,2.135
95.8.13.
94,5
89.0239
Negative Positive
28.82209
91.7419
91.4155
93.8222
94.4240
96.0133
94.7418
88.0254
Rec.-
98.8211
94.5
94.8124
95.1
87.0as2
82.019.3
87.51.6
94,71225
Negative Positive
36.5+3s.1 74.0
93.0+1.3
92.9427
94.4+a7
91.1+11
89.9428
91.5+11
91.0+24
F1-1
93.2409
93.24117
94.4
90.3a1x
87.91419
90.7422
91.7
Table 29; Precision, recall, and for SST-2, where 'b' refers RoBERTa-bass 'I' refers to RoBERTa-large.


Reviews:
[{"review_id": "3k1nt26G15", "reviewer": "Reviewer_RAHH", "summary": "", "questions": "* Line 485: The authors mention that the models were trained for 15 epochs. Do the authors use the checkpoint from the best performing epoch or do they use the last checkpoint? If the latter why?", "limitations": "", "rating": 2, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments and review! \n\nRegarding the InstructGPT comparison: InstructGPT incorporates reinforcement learning from human feedback (RLHF) to finetune the same model to follow a broad class of written instructions, thus enabling the model for cross-task generalization. However, we only use one written pattern for finetuning the model without human feedback (our different patterns correspond to different finetuned models), keeping the same pattern for training and testing instead of different instructions, and we use the same set of patterns for comparison between zero-shot and LabelDescTraining. Therefore, our approach differs significantly from InstructGPT.\n\nRegarding the hyperparameters for the ablations, we actually tuned hyperparameters (on 20NG) independently for each ablation setting. Please refer to Table 13 in Appendix A.3, where we explain our tuned parameters. We’ll be sure to add more details to the next version.\n\nRegarding a baseline trained on 20NG, thanks for your suggestion! We can easily add results of this model to the next version. \n\nRegarding choosing only 4 labels from 20NG for tuning, we agree that many other choices could be made here that could potentially improve our results overall. The four classes we chose have a similar topical granularity as the classes in widely used datasets like AGNews and Yahoo. If we had grouped labels or used more labels, we could potentially have found even better hyperparameters, but we did not consider different variations here. Thanks for the suggestion regarding moving in the direction of building general purpose classifiers. Some of the cited work we compare to, like Chu et al., does this. However, our paper instead starts with a given label set and seeks to build a classifier that is independent of the input text domain for that label set by using label descriptions.\n\nRegarding the results with text-davinci-003: We want to mention here that we did compare extensively to available SOTA results (Table 4) for all datasets. The reason we reported text-davinci-003 results is to show that our method not only works for MLM-style models like RoBERTa, but also works for autoregressive models like text-davinci-003. Although we have used limited data for text-davinci-003 mostly for cost reasons, the results present a similar pattern (label description training is better for topic datasets and highly comparable to other methods even against text-davinci-003). \n\nQuestion 1: We use the checkpoint from the best performing epoch selected by the dev set.\n\nThanks for your suggestions regarding the writing. We are evaluating every 24 steps (24 is the LabelDesc dataset size for 20NG, namely the number of steps per epoch). \n\nRegarding the use of the term LLM: we used the term LLM to represent any pretrained language model, and we’ll clarify this.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper is well-written and easy to understand.\n* The approach is straightforward and seems to improve over the pattern-verbalizer approach.\n* The idea to construct synthetic data using only the labels is interesting.\n* The authors have included ablation tests to demonstrate the robustness of their approach.", "reasons_to_reject": "* The results are neither surprising nor exciting. Zero-shot and few-shot learning with GPT-3 using prompting had still a large gap with supervised SotA models. On the other hand, InstructGPT [1] was much better than GPT-3 and the main reason was that it was trained to follow instructions (prompts). The proposed approach is very similar to the basic idea of InstructGPT. The authors fine-tune an MLM using patterns, which can be seen as instructions, and synthetic data. In effect, the model has learned to use these patterns and is therefore able to use this knowledge for classification.\n* The randomized and mismatched ablations are not very well designed. The authors use the same setting found during hyper-parameter tuning. However, the ablated versions have a more difficult task to solve. The randomized version has to learn new embeddings from scratch and the mismatched version has to learn a new meaning for each label. Thus the model may require more steps to adapt. I would be interested to see how the ablated versions would perform if the authors tuned the hyper-parameters on 20NG for each of these settings.\n* I believe the authors should have included one more baseline, i.e., the model trained on the 20NG dataset.\n* I am not entirely convinced by the choice of the authors to use only 4 labels from the 20NG dataset. For instance, similar labels could have been merged. In addition, the proposed approach could include more labels, e.g., from Wikipedia, which may or may not contain the labels of the downstream tasks, and create a larger dataset. This combined with the baseline I proposed above could lead to a general-purpose pattern-based classifier.\n* The results with InstructGPT (text-davinci-003) cannot be compared with the other results. It is understandable that the authors were able to only evaluate 1,000 samples to reduce the cost. I would expect however to compare these results with the other methods on the same set of samples.\n\n[1] https://arxiv.org/abs/2203.02155", "questions_to_authors": "* Line 485: The authors mention that the models were trained for 15 epochs. Do the authors use the checkpoint from the best performing epoch or do they use the last checkpoint? If the latter why?", "paper_topic_and_main_contributions": "This paper proposes a label-description training approach to improve the classification performance of pre-trained MLMs when using a pattern-verbalizer approach. Depending on the classification task, the authors create a dataset where the labels are the labels of the task and the inputs are based on descriptions, relevant topics, and/or simple patterns. They then fine-tune an MLM after converting the data using a pattern-verbalizer approach.\n\nTheir results show that fine-tuning the MLM on the created dataset is better and more robust than using the MLM out-of-the-box. The authors demonstrate the robustness of their approach with ablation tests where they use random verbalizers, shuffled verbalizers, and no verbalizers (and patterns) at all. Finally, they compare their approach with in-domain and out-of-domain few-shot learning approaches.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "* Lines 285 - 287: This part is a bit unclear. The authors mention both steps and epochs. I would them to evaluate every $n$ step or define the number of steps per epoch.\n* The Abstract and the Introduction begin by stating that LLMs have improved zero-shot text classification. Thus, they give the impression that the authors will improve the zero-shot capabilities of LLMs, but the authors focus on RoBERTa which cannot be considered an LLM."}}, {"review_id": "84abCRA46i", "reviewer": "Reviewer_S9DN", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review!\n\nRegarding practitioners’ understanding of the labels: A significant aim of this project is to demonstrate how simply curating a few training instances for each label can substantially enhance performance over zero-shot methods. Even though this does require some understanding from the practitioners, it's typically a simple task involving the gathering of synonyms or online dictionary definition. Furthermore, it only takes a few minutes to collect these instances. We've demonstrated that just a couple of such examples are adequate to create a more efficient classifier than a zero-shot one across all the data sets and domains we used.\nRegarding potential bias - although we mentioned this in the limitation section (line 616) our examples are largely very objective (e.g., a dictionary definition) which limits the tendency to result in additional bias.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper proposes a new way of constructing dataset which only consists of the description of the labels for LLMs on zero-shot text classification tasks. And this method is not strongly dependent on patterns and verbalizers. It is also hopefully to applied on other tasks.", "reasons_to_reject": "Though the method proposed by this paper could obviously improve the performance of LLMs on zero-shot text classification tasks, it's a simple method which has a strong rely on the practitioner’s understanding of the labels and usage situation. It also has a need of a pre-trained model which could produce some potential biases.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a new way to construct a dataset which could obviously improve the performance of LLMs on zero-shot text classification tasks. The data in the dataset constructed by this method only contains the description of the labels. In the topic classification and sentiment classification tasks in this paper, the model using this method outperform than the other baseline models with zero-shot.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "zm3t3JVRU7", "reviewer": "Reviewer_ZfZP", "summary": "", "questions": "Please clarify how the fine-tuning on LabelDesc data is exactly conducted. I'd consider updating my rating given the authors' response of this.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments! We’ll be sure to add more details (descriptions and figures) to the camera ready. Below we describe exactly how the training data is formatted during the training.\n\nFor topic classification, let's say, we are using the label “sports” (used both in AGNews and Yahoo), whose verbalizer is also chosen as “sports”. We illustrate a training example here. With the related term “racing” (see Table 1 in the paper) we create an input as follows: “racing Question: What is the topic of this article? Answer: [MASK].”, and the model will be trained to output the correct verbalizer “sports” using the masked language modeling head at the position of [MASK]. The related term “racing” could be replaced by the dictionary definition, other synonyms, or Wikipedia sentences. For instance, Table 1 presents the dictionary definition. \n\nFor topic classification in the inference time, assume that the original text input is “Need for carbon sink technologies Climate scientists tell a conference that greater efforts should be made to pull CO2 from the atmosphere.” from AGNews, the input to the model is as follows using pattern 1: “Need for carbon sink technologies Climate scientists tell a conference that greater efforts should be made to pull CO2 from the atmosphere. Question: What is the topic of this article? Answer: [MASK].”, and the model is expected to output the correct verbalizer “Tech” (gold label is “Sci/Tech”).\n\nFor sentiment classification, let's say we are curating an example for the label “Very Negative” (SST-5 or yelp-5 dataset), whose verbalizer is chosen as “terrible” for illustrations of short templates. We choose “awful” as a synonym term of the verbalizer, taking the template “It was t.” as an example, the input is as follows: “It was awful. Question: What is the sentiment of this text? Answer: [MASK].”, and the model will be trained to output the correct verbalizer “terrible”. \n\nDuring inference, we follow the same procedures as in topic classification. Assume that the original text input is “Effective but too-tepid biopic” from SST-5, with pattern 1 for SST-5, the input to the model is as follows “Effective but too-tepid biopic Question: What is the sentiment of this text? Answer: [MASK].”, and the model is expected to output the correct verbalizer “okay”, as the gold label is “Neutral” for this text example.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Novelty: The new paradigm for zero-shot text classification is interesting and novel based on my knowledge. This is a new attempt aiming to leverage generic label descriptions to boost zero-shot performance.\n* Comprehensive evaluation: The experiments are conducted on both topic and sentiment classification tasks which seem sufficient to me. There are also analytical studies regarding the sensitivity to prompt patterns/verbalizers and domain transfer ability.\n* Effectiveness: The new method is empirically effective across standard text classification benchmarks, outperforming previous standard zero-shot prompting and achieving similar performance to larger GPT3 models with RoBERTa-sized encoders. ", "reasons_to_reject": "* Presentation of the method: While the motivation and intuition is general clear with the label description training, I didn't find a very clear and concrete introduction of how the fine-tuning is exactly conducted (starting Line 271). Specifically, how would you train the model on class-representative keywords, short templates, and Wikipedia descriptions, respectively? What is the input format and the prediction target? This might be straightforward, but I'd appreciate a figure or at least some detailed descriptions with regard to this, which will help the readers understand the method better.", "questions_to_authors": "Please clarify how the fine-tuning on LabelDesc data is exactly conducted. I'd consider updating my rating given the authors' response of this.", "paper_topic_and_main_contributions": "The paper proposes a new paradigm for performing zero-shot text classification in the following way:\n* Several types of label descriptions are used create a more detailed description of each class label, including keywords, short sentences/templates, and Wikipedia definitions.\n* A pretrained text encoder model (e.g., RoBERTa) is fine-tuned on such small label description datasets.\n* The fine-tuned model is taken for inference on task-specific data in prompting format.\n\nThe authors evaluate on several topic and sentiment classification tasks and demonstrate that the proposed paradigm is able to outperform conventional zero-shot inference by large margins.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper creates a dataset of label descriptions which can help models improve zero-shot classification.

The idea is interesting and experiments sufficiently demonstrate the usefulness.
The reviewers feel that most of the paper is clear and easy to follow.

The baselines and comparison to existing methods could be further improved, following specific suggestions from the reviewers.
Certain parts of the method could be clarified, as pointed out by reviewer ZfZP.
Decision: Accept-Main