Gen-Z: Generative Zero-Shot Text Classification with Contextualized Label Descriptions
Authors: Sachin Kumar
Chan Young Park
Yulia Tsvetkov
Venue: ICLR 2024 poster (2024)
Keywords: zero-shot classification
prompting
generative classification
label descriptions
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We present a zero-shot prompting framework that outperforms or closely matches in-context learning for a variety of text classification tasks.

Abstract: Language model (LM) prompting—a popular paradigm for solving NLP tasks—has been shown to be susceptible to miscalibration and brittleness to slight prompt variations, caused by its discriminative prompting approach, i.e., predicting the label given the input. To address these issues, we propose Gen-Z—a generative prompting framework for zero-shot text classification. GEN-Z is generative, as it measures the LM likelihood of input text, conditioned on natural language descriptions of labels. The framework is multivariate, as label descriptions allow us to seamlessly integrate additional contextual information about the labels to improve task performance. On various standard classification benchmarks, with six open-source LM families, we show that zero-shot classification with simple contextualization of the data source of the evaluation set consistently outperforms both zero-shot and few-shot baselines while improving robustness to prompt variations. Further, our approach enables personalizing classification in a zero-shot manner by incorporating author, subject, or reader information in the label descriptions.

Published as a conference paper at ICLR 2024
GEN-Z: GENERATIVE ZERO-SHOT TEXT CLASSIFICA- TION WITH CONTEXTUALIZED LABEL DESCRIPTIONS
Sachin Kumar*
Chan Young Park Carnegie Mellon University Pittsburgh, PA chanyounêcs, cww.nneecccccc....mmmmmm cmu. edu
Yulia Tsvetkov
Allen Institute for AI
University of Washington Seattle, WA yu puliatsecs.
Seattle, WA
wwwbink@allenai..omm
wwwpaassssssssssinntttteeee washington. edu
ABSTRACT
Language model (LM) prompting popular paradigm for solving NLP tasks has been shown to be susceptible to miscalibration and brittleness to slight prompt variations, caused by its discriminative prompting approach, i.e., predicting the label given the input. To address these issues, we propose GEN-Z- generative prompting framework for zero-shot text classification. GEN-Z is generative, it measures the LM likelihood of input text, conditioned on natural language de- scriptions of labels. The framework is multivariate, as label descriptions allow us to seamlessly integrate additional contextual information about the labels to improve task performance. On various standard classification benchmarks, with six open-source LM families, we show that zero-shot classification with simple con- textualization of the data source of the evaluation set consistently outperforms both zero-shot and few-shot baselines while improving robustness to prompt variations. Further, our approach enables personalizing classification in zero-shot manner by incorporating author, subject, or reader information in the label descriptions
INTRODUCTION
Language models, trained only on raw text, have been shown to perform new tasks simply by conditioning on handful of demonstrations (Brown et al. 2020). However, how language models acquire this ability, known as in in-context learning (ICL), is subject of debate (Xie et al.
2022 Ahuja
al. 2023 Hahn & Goyal 2023 Zhang al. 2023 von Oswald et al. 2023 Wang et a.. 2023) with several studies suggesting that it merely serves as way to prime the model with the domain, concepts, or topics and the format of the target task (Min et al. 2022b Wang et al.) (023). Furthermore, ICL has been shown to be very sensitive to the choice of training examples, their order and format in the prompt (Lu et al. 2022, Sorensen et al. 2022 Sclar et al. 2024) requiring major human effort to achieve optimal performance. In this work, we ask, "If the right demonstrations are challenging to find and only serve to implicitly prime the model, can we achieve the same performance zero-shot prime the language model explicitly?"
We introduce GEN-Z, a robust zero-shot generative prompting framework for text classification (Figure which achieves results par with in-context learning with much better stability in performance. Our approach consists of two key ideas. First, most text classification methods follow a discriminative setup, which involves estimating the probability of the labels given the input, which can be sensitive to prompt or verbalizer variations. Instead, we use a generative setup, which involves estimating the probability generating the input given different labels, which has been shown to have better worst-case performance (Min et al. (202aa). Second, to prime the models to solve the task, we propose explicitly incorporate contextual information via expressive label descriptions. We first generate description for each label that captures various factors that can influence the label and then estimate the probability of generating the input text given the label description (e.g. "This Reddit post contains hate speech about race" for hate speech detection where the data source "Reddit" and the subject "race" are additional factors). Finally, to further reduce variance from different label
Part this work was done when Sachin was a PhD student at Carnegie Mellon University. We provide the code to reproduce our results
https: thub. com/ com/Sachi119/
generative-classsficctaa

---

Published as a conference paper at ICLR 2024
Discriminative Classifier
Multivariate Generative Classifier
Multivariate Generative Classifier with Label Descriptions
"The Recislt comment written woman emoowers the addressed woman:
X
Y
get girl!"
EMPOWERING
Generative Classifier
EMPOWERING
get girl!
U
W
W
Platform: Reddit
Reader: Female
Writer Femake
Figure 1: Illustration of the proposed zero-shot generative text classifier with label description and examples.
description phrasings, we propose to compute and aggregate the results across multiple paraphrases of the label descriptions.
We evaluate GEN-Z by conducting experiments with six open-source language model families (GPT2, OPT, Pythia, GPT-J, Llama, and Llama2) with models ranging from 125M 13B parameters on 19 semantic text classification tasks (comprising sentiment, topic, hate speech, and emotion classification). We show that incorporating readily available additional variables like text source, domain, subject, information of author, audience, and the addressee of the text, in our approach leads substantial improvements compared vanilla zero-shot baselines, strong retrieval based zero-shot methods, and performs par with heavily tuned in-context learning methods.
ZERO-SHOT GENERATIVE CLASSIFICATION WITH LABEL DESCRIPTIONS
This section describes our proposed method. First, motivate our zero-shot setup by drawing connections to interpretations of in-context learning as concept learning. We then give an overview of generative classification followed by an explanation of how we incorporate contextual information into the model using label descriptions.
2.1 IN-CONTEXT LEARNING IMPLICIT CONCEPT LEARNING
We seek to build probabilistic classifier, px)(x) that takes text x as input and predicts y the set of all labels. Language models trained predict the next token given the history have been shown to be able to perform classification tasks in-context without fine tuning (Brown et al. (020). Given i demonstrations ((x1, 1/), 99)...... ..., (xx: } and test example x, the label can be predicted using, ( V1. Vi..... Xg, Uk. x). In practice, the label is verbalized in natural language (e.g.. the words "negative" and "positive" for sentiment classification). Prior work (Xie et al.) 2022) has shown evidence that in-context learning implicitly performs Bayesian inference where this probability can written as the following marginalization,
PLM (y) X1, VI..... Nk. Uk. x)
p(u x1 .1...., Xk. Uk. x, ())((() B1..... Nk. Uk. x) x)
e
p(y|x. ()p(0(x), V1, .I.... X yk x)d0
where 0 E ② represents a concept or a topic variable, on in general, context required to solve the task. Following Wang et al. (2023), we also make simplifying assumption that the test example independent of the sampling of the demonstrations, and x. That is, the context variable acts like an approximate sufficient statistic for the posterior information related to the demonstrations. The variables are latent and the model is expected to implicitly figure out the context needed to solve the task from the given demonstrations, Intuitively, the concept mentioned in the demonstrations, that is, the LM softly predicts this concept. In this work, we take this formulation to the extreme by defining it as a Dirac delta distribution concentrated on the right concept, that is given the right concept 0*, set (..))
is independent of the demonstrations given
p(!)...) concentrates
2

---

To incorporate contextual information 0", we propose multivariate generative classification which generalizes it to more variables that might influence the generative process of the input text, expressing the generative probability of x as p(x u, ....... where 0 = u, c.... represent the additional factors. For example, to generate review, not only is the author influenced by the polarity but also Published as a conference paper at ICLR 2024
and zero everywhere else. For semantic text classification tasks, argue that the right concepts can be specified in natural language and hence, the demonstrations may not be necessary. The label prediction probability is thus reduced to,
( X1 y1...., Xk. yk. x) (LM(y|x,87 0")
This describes zero-shot inference contextualized on the concepts. In this work, we experiment with many different kinds of contexts and their influence on text classification performance such those describing the domain, source, author, or audience the input text among others ($3).
2.2 MULTIVARIATE GENERATIVE CLASSIFICATION
The approach discussed so far describes discriminative classifier, which predicts the label as ý = arg m p(ys(x). They are designed to distinguish the correct label among possible choices. generative classification framework reinterprets this objective using Bayes' rule and different factorization as
p(x, Us)
ÿ = arg max 9i
p(x)
arg max ((xx)(xxy.))pll 21
Here the denominator p(x) is independent of the label and can be ignored. Further, assuming equal prior probability of all labels, p(y)) can also be ignored making the classification objective arg max,, p( In a generative setup, we assume a label generated first (e.g., an author decides to write a negative review), and then the text (e.g., the negative review) is produced conditioned on the label. Prior work has shown evidence that generative classifiers can be more robust than the discriminative ones which may look for shorteuts to predict the label (Yogatama et al. 2017).
by the item they review, the medium where they write the review, their target audience, their writing style, and on. Similar context also be added discriminative classifier p(y) X, u, U,.... which is one of our baselines.
2.3 CONTEXTUALIZED LABEL DESCRIPTIONS
In practice, as introduced in (Brown al. 2020), language models can be used in zero-shot setup by computing (2)() x) or in our case, PTM ( u, U,...)). Here, 2(-) is referred to as a verbalizer which expresses the label in natural language form that meaningful probabilities computed. In this work, since the verbalizers are only concerned with the label, we refer to them as label descriptions. A simple example is "This is terrible." " and "This is amazing." for negative and positive label respectively. The choice of this description, however, can lead to large variance in the model performance with downstream classification performance can range from near perfect to near
be
chance (Liu et al. 2023 Holtzman et al.) 2021] Zhao et al. 2021).
To reduce this variance, we propose to use multiple variations the descriptions z. More formally, we modify the generative story the labels and other contextual variables generate label descriptions z which then inform the generation of the text (see Figure 1).
p(u: X, u, V....) x p(x, Fel (u, .....)
Σ
p(x, Vi, 24 2u
Σ
p(z zi, u, v,.. ......))(// U,.. .p(ussl, v....)
Here, [ig, u, v.... denotes the of all ways describe the label y; and the context in natural p(2 v....) measures the existence probability of the description Since each
language D
We show that 10 diverse paraphrases the description are sufficient to obtain good performance. NNoe that we are not measuring grammatical plausibility of description, hence measure p(2l-) using an LM is not appropriate in this setting.
3

---

Published as a conference paper at ICLR 2024
description is equally plausible to exist given the label y and other variables, we drop p(2g (2. M/. u, .....) (see Figure 1 Irigh).. Further, assuming independence of the label and the contextual factors and equal prior likelihood of all labels, we also drop p(y (u, v....) v, Hence, the first term in the summation can be reduced to Σ,, p(][2;), which is our inference objective, where we evaluate the probabilities using the conditional probabilities of the L.M. We compute this term for each label under consideration yi and predict the label which obtains the highest value. Notably, unlike common prompting scenarios, the label descriptions, 2i, are unique for each label y; being considered and can be specialized by adding any available information about the instance in natural language format. We refer to our approach as GEN-Z for Generative Zero-Shot Classification.
3 EXPERIMENTAL SETUP
Datasets, Models, and Label Descriptions We evaluate on 18 text classification datasets encom- passing diverse tasks, domains, and difficulty levels. These datasets include varying numbers of classes and attributes that can be used as additional context to improve the classification performance. We consider all contexts that were provided with each dataset. We consider the following tasks divided in to two groups: (1) sentiment, topic, and hate speech detection which in addition to the input text are accompanied by information about the domain, source or subject of the input text. For example, hate speech datasets which contain information about the source (such as Reddit or Twitter) and the subject of hate (such as national origin or race); and (2) politeness, and empowerment prediction which are pragmatic tasks that depend on social variables such as demographic information of the author, addressee, or the reader (such as gender, age, educational background, etc.). Table use. We measure performance using
iin the appendix summarizes the details of each dataset publicly available validation or test sets, without using the training data at all. We experiment with the six classes of open- models: GPT2 (Small, Medium, Large, and XL) (Radford et al. 2019), OPT (Zhang et al.] (002)) and 2.7B), Pythia (Biderman et al.] 2023) (1.4B, 2.8B and 6.7B),
GPT-J (6B) (Wang & Komatsuzaki 2021), Llama (Touvron al. (023a)) (7B and 13B) and Llama
2 (Touvron et al. 2023b) (7B and 13B) All these models are pretrained on only raw text without additional fine-tuning on supervised datasets
For each task, we manually write one minimal label description per label using a template (see complete list in Table 5). We then generate 20 paraphrases of each label description by querying C This process needs to be done only once for each task and, in practice, any paraphrasing model can employed. We further manually verify the correctness of each paraphrase. For each dataset, we run the evaluation ten times where in each run we subsample 1 S n S 10 paraphrases from this set. We evaluate all methods using macro-F1 score and report mean and standard deviation across these runs.
Baselines We compare GEN-Z with the following zero-shot baselines.
Diseriminative methods predict the label using Σx, p(2 We consider three versions of this baseline. DISC-SINGLE-NC predicts the label with no context (the context information is removed from 2,) and only one description considered. This is the simplest zero-shot setup that most prior work considers canonical. DISC-SINGLE predicts the label using p( 2; x) where corresponds only one deseription. DISC-MULTIPLE predicts the label using Σ,, a, p(2. ((24) }x) which is the
discriminative version of our proposed method. For the last two baselines, we further have three
"The true prior probability of any label is unlikely to depend on the contextual factors like the domain of the text, or the personal attributes of the user reading, writing, or being described in the text. 'While we make this assumption for simplification, future work may consider non-uniform priors further improve performance.
"We do not report results with 70B sized models to due to its high computational requirements for the scale of our experiments. While quantization (Dettmers et al. 2023) approaches have heen proposed run models of this scale on consumer hardware, in our initial exploration such approaches vastly underperformed 16-bit versions for our experiments. Further, our budget prohibits us from experimenting with closed-source models like GPT3 which according to Lyu et al. (2023) can cost more than 4500 USD for the scale of our experiments. We leave these explorations for future work.
7 IInstruction- -uned models have shown to also perform well in-context but they are trained primarily as discriminative classifiers and thus cannot be used for generative classification making comparisons unfair.
We used the free tier of ChatGPT for this purpose: https:://chatt openai. ctm/ chat.
4

---

Published as a conference paper at ICLR 2024
Table 1: Zero-shot Macro-Fl with GPT-J (6B). We report average over 10 runs. Our proposed approach is GEN-Z. More results are provided in Appendix C with their initials.
The baseline names are shortened
DS-N
DS
DM
DP-N
DP
DPM
GS-N
GS
GEN-Z
SST2
(9.5(1.2)
65.9
43.6(1.0)
24.5
(5.7
(2.3
80.0
87.1
(0.7
SSTS
18.6(a.7)
24.2
25.1
26.2.om
29.8:o.8)
35.6(0.3)
34.8
36.0(1.2)
(0.9(0.5)
Yelp2
68.1
64.2
76.7
70.5
70.2
(5.9
76.1
85.4(a.7)
(9.9
Yelp5
14.0
26.2
25.610.3)
30.5;1.1
28.5(1.4)
32.0
35.5
38.0
42.0
(0.7
(3.3
51.3(0.5)
72.3(1.0)
70.6
74.6(0.5)
76.0
84.2
(..0)
CR
(2.3
(7.1
51.8(11)
60.3/2.1)
66.4(2.9)
66.7
72.9(1.7
83.8(1.5)
(7.0
Tweet3
36.0(u
45.5(0.4)
31.9(o.1
39.4..44
39.7
43.0(o.1)
42.2(003)
(2.2(0.3)
(l.1
FP
36.6(2.3)
32.3(2.0)
25.2(0.5)
38.7(1.9)
39.8;3.2)
(1.4
44.7(1.8)
48.2(2.4)
$2.9
PS
38.0
(2.9
30.3
35.83344
33.8
21.3
39.2(3.3)
38.4(1.2)
42.4
AGNews
34
34.8
37.9(z.)
54.6(0.5)
54.6(0.5)
(2.0
64.9(o.4)
(4.9().4)
(7.0 (0.1)
DBPedia
(2.5(u.5)
39.4
32.8(0.4)
61.7(0.5)
66.6(0.8
(8.9
71.7
71.7
80.
HS18
(7.4(0.5)
14.7
10.1
37.2(1.0)
38.7
29.9/0.3)
50.2
55.8
(2.6(
Dihos (NO)
50.8(a.0)
50.8(a.0)
63.2
54.9(2.1)
54.9/2.1)
55.7(1.))
50.4;3.0)
50.4(3.0)
56.3(0.8)
Bthos (SO)
33.8(4.7)
33.8
20.4
52.5
52.5:5.4)
55.3
52.7
$2.7
62.3
Ethos (Race)
27.4(a.5)
27.4(a.8)
15.5
54.5(4.3)
54.5/4.)
(0.9
56.1
(6.1
60.5
(Religion)
48.2(a.9)
48.2
42.5(1.8)
62.3(2.8)
62.3/2.8)
62.8(1.))
62.9(a.7)
(2.9(a.7)
70.1
Emotions
24.9(0.4)
23.5(0.8)
29.0(0.3)
31A(1.0)
31.8(1.1)
37.8(0.3)
30.3
30.3
(2.7
versions of each which differ how the context provided (only in the label, before the input, and before the input as an instruction; more details in (Appenddxx B We report results with the first version the main paper as it performs the best of the three.).
Calibrated discriminative methods use p(2; (z)pp/ppp((pp (NULL) (NULL) for label inference (we use BOS token for the corresponding LMs as NULL). Since language model probabilities can be poorly calibrated and suffer from competition between different label descriptions with the same meaning, this method relies on pointwise mutual information (PMI) between æ and to make a predic- tion (Holtzman et al. 2021). We again consider three versions of this setup: without context (DISC-PMI-NC), with context but only one label description (Disc-PMII and with context and multiple label descriptions
The last is the calibrated discriminative
version of our proposed method.
Generative baselines predicts the label using p(x[24) that is using only one label description. We consider two versions of this baseline, one without context (GEN-SINGLE-NC) and one with context (GEN-SINGLE) in the label description. Both of these are an ablation of our proposed method. The former method (without context) also describes the method proposed in Min et al. (2022a).
In addition, show comparisons with the following baselines which incorporate context implicitly either via shw-shot examples or using retrieval based techniques on unlabeled data. For these basclines, experiment with 8- and 16-shot settings and report the best of the two.
ICL describes in-context learning baselines. We consider four versions of this baseline: (a) ICL- DISC, a simple discriminative method which compares probabilities of label descriptions, (b) ICL-GEN, generative baseline with the exact same setup (Min et al. 2022a). (c) ICL-PMI which calibrates the probabilities same as DISC-PMI, (d) ICL-DC which another discriminative calibration method introduced in |Fi et al. (2023b).
Z-ICL (Lyu et al. 2023) describes a demoossttmmottttiioonnn based setup where unlabeled texts are sampled from a corpus and assigned random labels. This setup is zero-shot but still requires access to a corpus. For this setup, we reproduce the setup proposed in and report both discriminative and generative results.
RESULTS
We categorize the results into two groups: domain-aware classification, which considers the domain of the text as an additional factor, and personalized classification, which includes personal attributes of writers and readers additional factors.
5

---

Published as a conference paper at ICLR 2024
Table 2: Best of 8/16 shot basclines vs Gen-Z (zero-shot). We report average over seeds.
ICL (Dise)
ICL (CC)
ICL (DC)
ICL (Gen)
Z-ICL (Disc)
Z-ICL (Gen)
GEN-Z (Ours)
SST2
9..0
90.8(a.2)
94.0
88.8
82.6(o.2)
82.6(0.2)
(1.7(0.2)
CR
81.4(6.9)
86.5
(7.0(4.0)
84,4(2.8)
78.8 78.8;0.4;
80.1
(7.0
MR
(3.7
91.3
93.1 (0.5)
84.0/6.8)
81.0 81.0
81.9(a.)
87.0(0.2)
SST5
42.9
40.8:5.4)
40.3
42.9
30.9/o.3
38.7
40.9(o.)
FP
(6.4(6.9)
46.7:4.2)
61.6(3.8)
43.3
44.9(3.0)
51.1
(2.9
PS
26.6(6.1)
25.5(5.2)
31.4(3.0)
39.8(2.1)
39.5
43.9
42.4
AGNews
68.4(9.9)
76.8,7.2)
(3.5
72.3(3.2)
67.2(1.3)
75.2
(2.0
DBpedia
83.5(3.0)
90.6;1.7)
92.4(1.2)
79.9(3.8)
61.3
74.9
80.1
HS18
51.5(4.7)
41.6/8.0)
57.3/2.8)
45.0
43.4
51.1
62.6(0.3)
(Religion)
30.7(14.3)
28.0(13.8)
43.8
67.9(.8)
51.0/2.6)
51.0(2.)
70.1
E (NO)
(3.1
18.2/2.1)
(0.7
37.6(3.5)
37.6
26.2
56.3
(Race)
36.4(11.8)
(4.7(17.4)
51.4(6.)
49.1
46.3(1.2)
39.7(a.))
60.5
4.1 DOMAIN-AWARE CLASSIFICATION
Table shows the comparison of the performance of different zero-shot methods on sentiment, topic, emotion, and hate speech classification for GPT-J. Table 2 shows comparisons of GEN-Z ZZ with the few-shot baselines. The remaining results are reported Apppendix C
We find that GEN-Z Z overall outperforms all baselines approaches in the zero-shot setting. We do not see a clear trend in the simple discriminative baselines (DisC-SIMPLEE-NC DISC-SIMPLE, and DISC-MUL.TIPLE) where adding contexts and multiple descriptions sometimes help and sometimes hurts performance. In the calibrated discriminative baselines, the trends become clearer. The no-context version outperforms simple discriminative baselines as it accounts for surface form competition. Adding context also helps but adding multiple label descriptions does not always improve performance. We see this trend also in full results where range the number of label descriptions from to 10. The generative baseline without context and a single description (Min) et al. 2022a GEN-SIMPLE-NCI outperforms most discriminative approaches and adding context leads to even more improvements confirming the efficacy of our method.
Furthermore, GEN-Z in zero-shot setting is either best second best performing method when compared to strong few-shot baselines on sentiment and hate-speech detection (Table I). One particular dataset where our method lags behind Dbpedia topic classification where the label set consists of 14 classes. Our qualitative analysis reveals that most errors made by our approach correspond to three classes with semantic overlap (Album, Film, Written Work) which given our simplistic label descriptions make it difficult for the model to distinguish. This requires further investigations into the specificity of the descriptions which we leave for future work. Finally, compared to all baselines GFN-Z. shows the smallest variance in performance due to prompt selection by aggregating over multiple prompt paraphrases, whereas few-shot basclines exhibit large deviations.
We additionally conduct ablation studies to assess the impact of each proposed component on performance. While some these ablations we used for baseline comparisons in Table I here we give them more thorough treatment comparing performance across multiple model sizes.
Effect of number of label descriptions. In this ablation, we vary the number of label descriptions which the aggregation is perform from 1=1 1=10 and observe the change in performance. For each l, we do this evaluation 10 times and report the averaged mean and standard deviation across all 17 tasks. We find that in the generative classification settings, in the majority of cases, increasing the number of label descriptions improves the model performance highlighting the utility of this approach. Further, we observe that the performance starts stabilize between k=6 and k=100 which suggests that not many descriptions are required overall. In contrast, for discriminative baselines in all three versions we considered, we observe no clear trend as increasing & often results in a decrease in performance.
Effect of additional variables. To measure the effect of provided contextual information (do- mains, subject, data source), conduct ablation by modifying the label description to exclude this information across different number of label descriptions (similar to GEN-SIMPLE SIMPLE-NC and MISC-SIMPEE-NC) We report the full results in Appendix C We observe significant drop in

---

Published as a conference paper at ICLR 2024
473
Io triendlyyall
Figure 2:
Full results for domain aware classification for GEN-Z. The x-axis shows number of label descriptions per label and the y-axis indicates the average accuracy across all the tasks except (Politeness and Empowerment). We conduct a thorough analysis this setup as discussed in (section 4) removing domain information from the descriptions, different aggregation strategies as well as evaluating on different model sizes and families.
the performance across all tasks if we remove the domain or data source information including our method as well as the baselines. This drop is more significant in larger models. We hypothesize that specifying the domain information helps prime the model probabilities to the right distributional landscape allowing more meaningful comparisons between probabilities assigned to different labels. Further, we hypothesize that in a generative classification setting, the label followed by the input text, resemble natural data found in pretraining whereas it is difficult to specify this information in discriminative setup.
Effect of model size. We measure if the presented results holds across model scales. We repeat the same experiment across 14 more models with size ranging from 125M to 13B. We find that with GEN-Z, across reasonably large models, going larger improves performance on average. We see substantial improvement from GPT2-M to L to XL, and Pythia 1.4B to 2.7B and 6.7B to 12B The trend is reversed for the Llama2 models where the 13B models performs slightly worse overall and warrants further investigation.
Ablation on aggregation strategy In GEN-Z, we aggregate the probabilities obtained using different label descriptions by simply summing them (which is the same as their arithmetic mean, for comparison purposes). This aggregation is theoretically grounded in the probabilistic framework we design (Figure I). Prior work has considered several other aggregation strategies for ensembling model outputs that we compare with in this ablation. We compare against geometric mean (or arithmetic mean of the log probabilities, the most common way to aggregate model outputs) and harmonic mean. We find that in our proposed generative setup, the performance across three aggregation strategies is
The performance is strictly comparahle across model families due differences in pretraining corpora.
7

---

Published as a conference paper at ICLR 2024
Aut. \Ad.
No
Yes
Age \ Ed.
No
Yes
No
81.1 84,8
No
80.1
81.4
(1.43)
(1.83)
(0.30)
(0.21)
Yes
81.8
85.0
Yes
82.2
83.3
(1.07) (2.42)
(0.32) (0.24)
(a) Empowerment (FI),
(b) Politeness (Accuracy)
Table 3: Personalized classification results on GPT2-Large with our model. Each cell repre- sents whether the demographic attribute was used in the label description or not. Aut.=Author, Ad.=Addressee, Ed.=Education.
largely similar, arithmetic mean outperforming the other two slightly overall with harmonic mean winning out in smaller models. We hypothesize that this effect is due to harmonic mean's 's property of ignoring outliers. Future work may analyze this strategy in-depth. For the discriminative setup, arithmetic mean almost always performs the best. Peculiarly, harmonic means shows sharp decrease in performance with increasing descriptions and warrants further investigation which we leave to future work.
4.2 PERSONALIZED CLASSIFICATION
In this setup, we evaluate our proposed approach on two datasets where personal information about the author, the addressee, even the audience may affect the prediction. We experiment with two tasks: (1) empowerment prediction (Njoo et al.] (023) where given Reddit comment, the goal is to predict whether it empowers or disempowers (or is condescending to) the addressee the comment. We use the author's and the addressee's 's gender in this tas (2) Politeness prediction (Pei & Jurgens 2023) where given an email snippet the goal to predict whether it polite or not. We again consider binary labels. What is considered polite may vary with the reader dependent dataset consists of information about the annotator's age, gender, race, and educational background. We focus age and educational background as they were the primary delineators of variation measured by the authors. That is given the author's age and educational background, we predict the perceived politeness of the text and sum their probabilities to make the final predictions. We do aggregate these probabilities over each possible value of age and educational background but rather only the ones reported in the test set. The results for both datasets for GPT2-Large are reported in Table 3 We only report the results for our proposed approach with varying number of personal attributes considered, as discriminative models performed poorly in this setup (<50% accuracy across both tasks). We find that for both test sets, personalizing the predictions with demographic variables helps improve performance. For empowerment prediction, the gender of the addressee, and politeness, the age of the annotator affect the performance more than the other variables. The latter is consistent with prior studies that show cultural differences politeness across different age groups (Pei & Jurgens 2023).
cultural f'cctors. This
RELATED WORK
In-context learning In-context learning (ICL) is the standard paradigm for prompting LMs to perform tasks (Brown et al. 2020 Liu ct al. 2023).Much Mucch recent work has been done to understand why it works and how it can be improved. For example, Xie et al. (2022); Wang et al. (203)): ) Dai jet al. (2023) have argued that it implements general-purposs learning mechanisms such as Bayesian inference gradient descent. Min al. (2022b) showed that for classification tasks, the input-label pairing format plays the most crucial role in ICL. We build on these findings and develop a zero-shot inference approach. While in-context learning with more example has usually performed than zero- shot inference, it comes at the cost of more token consumption and may hit the context length limit when the input and output text are long. The choice of demonstrations can lead to high variance in the model performance (Zhao et al. 2021 Fei et al. [023a] Han et al.) 2023) and prior work has investigated various demonstration selection- and ordering strategies boost performance LLu 2022).
WeW use binary gender here; the evaluation set does not contain any other information
The results for other models can be found in Appendix C
8

---

Published as a conference paper at ICLR 2024
In this work, we show that the zero-shot setting is underexplored and can surpass in-context learning for text classification tasks.
Recent work has also studied instruction following in models, either directly on a pretrained language model or by fine-tuning it to follow instructions using a collection of NLP tasks framed as instruction following tasks (Wei et al. 2022). Instructions and few-shot learning also be used together. Again, depending on how instructions are phrased, however, can significantly alter the model outputs, even in instruction fine-tuned models (Sun et al. (023). In contrast, several studies have also developed prompt engineering techniques, that is creating sequence of prefix tokens or prompts that increase the probability of getting desired output given input (Liu et al. 2023). These techniques rely on available training data for each task. In this work, we focus on a zero-shot prompting setup operating in setting where no training data for customizing classification models is available.
Discriminative versus Generative Classification Text classification studies with prompting have primarily focused on discriminative classification, which focuses on constructing input prompts that get prepended to each imput text to predict the classification label. That is conditioning to generate the output. Generative or noisy channel models (Brown et al. 1993) have been previously investigated for various NI.P tasks, such machine translation Yaamada & Knight] [001] Yeel et al. (219)) and question answering (Lewis & Fan 2019). Prior work has empirically demonstrated that generative models are more robust distribution shift in text classification than diseriminative models (Yogatama et al. 2017). Recently, Min et al. (022aa) explored the use of a generative model with prompting, leveraging pretrained language models for various text classification tasks. In this work, we build a multivariate generative classification by incorporating label descriptions. These descriptions capture various contextual information associated with each example, allowing for effective priming and customization of the classifier.
the input
Social and personal factors in NLP Machine learning systems have been shown to reflect and amplify social prejudices in human-written text, resulting in systemic biases in performance towards specific demographic groups (Mehrabi et al.] 2021). Such classifiers learn spurious correlations between the label and the demographic information reflected in text either explicitly through their mentions in the text (such as names, sexuality, and race among others) or their writing style. These issues are exacerbated through annotation artifacts (Sap al.) [219]] 2022) or unbalanced datasets (Kir) itchenko & Mohammad, 2018). Various solutions proposed in the literature aim to learn models that are fair to all demographics using methods like adversarial learning (Han et al. 2021 b) and distributionally robust optimization (Zhou et al. 2021). A distinct but closely related motivation to- wards developing such solutions is user privacy mrdvayy-mddels models should never use any personally identifiable attributes to make any predictions as it could lead to unintended negative consequences (Elazar & Goldberg 2018). Ravfogel et al. 2020; 2022) propose methods to scrub demographic information from model representations given trained model with little loss in model accuracy.
In contrast, few studies have shown that incorporating factors such as gender, age, region, or country of the authors features can improve text classification performance (Volkova et al. 2013 Hovy
2013] Yang & Eisenstein 2017] Lynn et al. 2017]] Huang & Paul 2019 Ostapenko et al. 2022). Most
of these studies are based on the assumption that social and personal factors are causally related to both the writing style and the target label. As a result, they treat classification as domain adaptation problem in which demographic attributes divide the data distribution into different domains.
CONCLUSIONS
We introduce GEN-Z, a robust generative zero-shot text classification framework that seamlessly incorporates contextual information beyond the input text itself. GEN-Z leverages LM likelihood of generating the input text based on various label descriptions that reflect context, enabling more robust predictions than discriminative approaches. We evaluate our framework across two task categories: domain-awar classification and personalized classification, covering 19 diverse text classification datasets with varying tasks, domains, and difficulty levels, alongside multiple label description paraphrases. Our experiments show that GEN-Z consistently improves classification performance over zero-shot baselines and performs on par with strong few- baselines. Further, we show that this approach allows personalizing predictions by incorporating contextual information from label descriptions.

---

Published as a conference paper at ICLR 2024
LIMITATIONS
The generalizability of our paper's findings to languages other than English may be limited since all the datasets used in our study are exclusively English. We make simplifying assumptions about prior probabilities of labels and independence of labels and contextual factors, which may not always hold in practice. We acknowledge that certain demographic attributes in our work may not fully represent the entire population. For example, due to data availability, experiments with binary gender (male/female) for user information, despite the existence of diverse genders. Additionally, the definition and categorization of social attributes in the datasets used in our experiments might predominantly reflect Western-centric perspectives, as the majority of the work involved in designing and creating such datasets aligns with Western-centree viewpoints. Lastly, while our experiments encompass diverse range of text classification tasks, we have not evaluated its performance on other kinds of tasks like sentence pair classification, question answering, etc. We leave these evaluations for future wrok.
only conducted
ETHICS STATEMENT
Personalization presents complex ethical considerations, with both benefits and potential risks. On the one hand, models tailored to specific settings or groups can yield positive outcomes. However, these personalized models may inadvertently reinforce biases result in discriminatory behavior their performance is uneven across different groups. Moreover, privacy concerns arise as end users may be reluctant to have certain attributes or personal information, such as their sexual orientation or religion, considered by the model. We believe our approach of using label description can mitigate personalization methods. By
such ethical concerns, particularly in comparison to embedding- employing interpretable user information through label descriptions, our method fosters transparency and controllability throughout the entire personalization process. This mitigates potential issues related to privacy and allows users to have insight into how their information is used. Nevertheless, it important to acknowledge potential cases of misuse, where individuals intentionally modify their user attributes to game the model and achieve desired labels. Such scenarios highlight the need for future research on mitigating abuse and maintaining the integrity of the personalization framework.
ACKNOWLEDGEMENTT
Part of this work was done when S.K. PhD Student at Carnegie Mellon University and was supported by Google PhD Fellowship. This research is also supported in part by the Office of the Director of National Intelligence (ODNI). Intelligence Advanced Research Projects Activity (IARPA), via the HIATUS Program contract #2022 22072200004. This material is also funded by the DARPA Grant under Contract No. HR001 12000124 We also gratefully acknowledge support from NSF CAREER Grant No. IIS2142739, NSF Grants No. IIS2125201, IIS2203097, and the Alfred Sloan Foundation Fellowship. The views and conclusions contained herein are those the authors and should not be interpreted necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, or the U.S. Government. The U.S. Govermment is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
REFERENCES
Kabir Ahuja, Madhur Panwar, and Navin Goyal. In-context learning through the bayesian prism, 2023.
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O' O'Brien, Eric Hal- lahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, al. Pythia: A suite for analyzing large language models across training and scaling. arXiv preprint arXiv:2304.00333,, 2023.
Peter F. Brown, Stephen A. Della Pietra, Vincent J. Della Pietra, and Robert L. Mercer. The mathematics of statistical machine translation: Parameter estimation. Computational Linguistics, 19(2):263-311, 1993.
2003
10

---

Published as a conference paper at ICLR 2024
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL WtRthttps: http:://ccpppceeeinn..... neur pdf
Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, and Furu Wei. Why can GPT learn in-context? language models secretly perform gradient descent as meta-optimizers In Findings of the Association for Computational Linguisties: ACL 2023, pp. 4005-4019, Toronto, Canada, July 2023. Asociation for Computational Linguisties. URL httphttt//acaaanhhooloo org/ 2tt/22233.r0 www.fiiininnniim 247
Ona de Gibert, Naiara Perez, Aitor García-Pablos. and Montse Cuadros. Hate speech dataset from white supremacy forum. In Proceedings of the 2nd Workshop Abusive Language Online (ALW2), pp. 11 11-20, Brussels, Belgium, October 2018. Association for Computational Linguistics. doi: 10. 10.18653/v1/W 1115//128 URL httlhtts////ccaattooooog/ wwwp//oogyyoor//--111100
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized Ilms, 2023.
Yanai Elazar and Yoav Goldberg, Adversarial removal of demographic attributes from text data. In Proceedings of the 2018 Conference on Empirical Methods Natural Language Processing, pp. 11-21, Brussels, Belgium, October-November 2018, Association for Computational Linguistics. doi: 18.53//3/v 1/B///////11//// 1002. URL htets//tt////lllatthoogyyy org.:lloyyyooroooooooo wrg/
Yu Fei, Yifan Hou, Zeming Chen, and Antoine Bosselut. Mitigating label biases for in-context learning. ArXiv, abs/2305.19148, 2023a. /tthttss:///pp http:////appssmmaiiiccoooooo/// semanticscoolar wrwp:/rpss//o/roCC//o/o
258967265
Yu Fei, Yifan Hou, Zeming Chen, and Antoine Bosselut. Mitigating label biases for in-context learning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 14014-14031, Toronto, Canada, July 2023b. Association for Computational Linguisties. httphh://acaaattolooo.oooo
acl long. 783
Michael Hahn and Navin Goyal. A theory of emergent in-context learning as implicit structure induction, 2023.
Xudong Han, Timothy Baldwin, and Trevor Cohn. Decoupling adversarial training for fair NLP. In Findings of the Association for Computational Linguistics: ACI-IJCNIP 2021, pp. 471- 477, Online, August 2021a. Association for Computational Linguistics. doi: 10. 00.16533/1/2021. findings-acl.44 acl.41. URL / wwwpo/e1........ooo findings-acl.41
Xudong Han, Timothy Baldwin, and Trevor Cohn. Diverse adversaries for mitigating bias in training. In Proceedings the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 2760-2765, Online, April 2021b. Association for Computational Linguistics. doi: 10. (0.188533v1/2021 wat.222.eealllmmn eacl-main 239. URL htt.httts:///alaathholoo org/2021. www.laacalmmlmmme 239
Zhixiong Han, Yaru Hao, Li Dong, Yutao Sun, and Furu Wei. Prototypical calibration for few-shot learning of language models. In The Eleventh International Conference Learning Representa- tions, 2023. URL http:tttss////eeeiiieee wwwprrewwwwieeeeefrfreeeee met/fooruniddmuuss9FFFAa
Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoye.. Surface form competi- tion: Why the highest probability answer isn't 't always right. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing. pp. 7038-705 Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.1100.... 18653/ v1/2021. mamlp--minn55e WtRLhttps://// http://sss/aaaaaaoooooooo.o.o//
amnlp-main.56. 564

---

Published as a conference paper at ICLR 2024
Dirk Hovy. Demographic factors improve classification performance. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joins Conference Natural Language Processing ( (Volume 1: Long Papers), pp. 752-762, Beijing, China, July 2013. Association for Computational Linguistics. doi: 10.31 1..1155/1/P15--007 URL
https://aalaanthologg... htt.aaccaaanttoooo.ooo////11111.
Minqing Hu and Bing Liu. Mining and summarizing customer reviews. Proceedings of the Tenth ACM SIGKDD International Conference Knowledge Discovery and Data Mining, KDD '04, pp. 168-177, New York, NY. USA, 2004. Association for Computing Machinery. ISBN 13881.. doi: 011145/1014452.00140777 /Rt/httss httt::::/doo.rrgg/ 18. / 00155//111155...0 1014073
Xiaolei Huang and Michael Paul. Neural user factor adaptation for text classification: Learning to generalize across author demographics. In Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 20/9), pp. 136-146, Minneapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: 10. 10.18653/11/511-1005 URL https hww.aaathooooyorrr/////.
Svetlana Kiritchenko and Saif Mohammad. Examining gender and race bias in two hundred sentiment analysis systems. In Proceedings of the Seventh Joint Conference on Lexical and Computational Se- mantics, pp. 43-53, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. www.aaattooooooorr////0---00
doi: 10. 18653v /S////1110SSS
Mike Lewis and Angela Fan. Generative question answering: Learning to answer the whole question. In International Conference on Learning Representations, 2019.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Comput. Surv. 55(9), jan 2023, ISSN 0360-0300. doi: 10. 145/545/5560815. URL https://doi.co org/ 10. 1o..o////../////..11 0111////355888
Daming Lu. daminglu125 at SemEval-2022 task 2: Using BERT and LSTM to text classification. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2002), pp. 186-189 Seattle, United States, July 2022. Association for Computational Linguistics. doi:
semeval-1.1 10.18663/v1/200222eeeeeaaa- 1.22. URL UBL/https://aalaathooo.g. wwwp:///.oooo.oo//2222.. wttp:////2222.seeell111111111111 -2
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8086-8098, Dublin, Ireland, May 2022. Association for Computational Linguistics. 1/122/
doi:
BtRLLhhtps:000 / /tttt::s//aclannhoooggg..oo wrt.:/rgg/////// acl- ac1- 556
Veronica Lynn, Youngseo Son, Vivek Kulkarni, Niranjan Balasubramanian, and H. Andrew Schwartz. Human centered NLP with user-factor adaptation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 1146-1155. Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18653/11/D17-111 URL https://aclanthhooog 0
Xinxi Lyu. Sewon Min, Iz Beltagy, Luke Zettlemoyer, and Hannanch Hajishirzi. Z-ICL: Zero-shot in-context learning with sscudo-demonstrattoon. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 2304-2317, Toronto, Canada, July 2023. Association for Computational Linguistics. URL UtRl/https://acllntoooll
org/ wrrg/2223.c0m awl.aclllool long. 129
Malo, A. Sinha, P. Korhonen, Wallenius, and Takala. Good debt or bad debt: Detecting semantic orientations in economic texts. Journal of the Association for Information Science and Technology, 65, 2014.
Ninarch Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey on bias and fairness in machine learning. ACM computing surveys (CSUR), 54(6):1-35, 2021.
Sewon Min, Mike Lewis, Hannanch Hajishirzi, and Luke Zettlemoyer. Noisy channel language model prompting for few-shot text classification. In Proceedings of the 60th Annual Meeting of
12

---

Published as a conference paper at ICLR 2024
the Association for Computational Linguistics (Volume I: Long Papers), pp. 5316 5330, Dublin, Ireland, May 2022a. Association for Computational Linguistics. doi: 10.16653/11/2222.cc1ooor 365. URL
acl acl-long. 365
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 11048-1 1064, Abu Dhabi, United Arab Emirates, December 2022b. Association for Computational Linguistics. URL https://aclanthologgy
wwwgg//02..ecm www.eemnnmmae... main. 759
Ioannis Mollas, Zoe Chrysopoulou, Stamatis Karlos, and Grigorios Tsoumakas. ETHOS: multi- label hate speech detection dataset. Complex & Intelligent Systems, January 2022. ISSN 2198-6053, doi: 10.1007/:40747-021-00000
URL httppp:///iiorrrggggooo 000.0/00000000000000070000000000
Lucille Njoo, Chan Young Park, Octavia Stappart, Marvin Thielk, Yi Chu, and Yulia Tsvetkov. Talkup: A novel dataset paving the way for understanding empowering language, 2023.
Alissa Ostapenko, Shuly Wintner, Melinda Fricke, and Yulia Tsvetkov. Speaker information can guide models better inductive biases: case study on predicting code-switching. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3853-3867, Dublin, Ireland, May 2022. Association for Computational Linguisiics doi: 10.18653/v 1/2552////022.acll// 2on..22.....11..26...6. URL /RRLntttss//aalaathoo.oo
2022 www.......looo.........
Bo Pang and Lillian I Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL (O5), pp. 1 115-124, Ann Arbor, Michigan, June 2005, Association for Computational Linguistics. doi: 10.3115/1219840 11111/12.884401..885... 1219855, URL
org/P05- 1015
Jiaxin Pei and David Jurgens. When do annotator demographics matter? measuring the influence of annotator demographics with the popquorn dataset, 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 1(8):9, 2019.
Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. Null it out: Guarding protected attributes by iterative nullspace projection. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7237-7256, Online, July 2020. Association for Computational Linguistics. doi: 10.1865//1120200.ccl awt.ma0....mmmmi..6777....7 URL https //aclanthology.oo
ww122..clmmain.... 647
Shauli Ravfogel, Francisco Vargas, Yoav Goldberg, and Ryan Cotterell. Adversarial concept erasure in kernel space. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 6034-6055, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/1//0222ccmm 1/2022.emmlpp-ppp... main.405. URL https://aclanttoooggy.oo org gaarg22222 wtw.222.emmppmma..mm. emn-p-main.400
Sara Rosenthal, Noura Farra, and Preslav Nakov. SemEval-2017 task 4: Sentiment analysis in Twitter. In Proceedings of the 1 11th International Workshop on Semantic Evaluation (SemEval-2011 pp. 502-518, Vancouver, Canada, August 2017. Association for Computational Linguisties. doi: 10. 10.18653/ URL/
Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. The risk of racial bias in hate speech detection. In Proceedings of the 57th Annual Meeting of the Association for Compu- fational Linguistics, pp. 1668-1678, Florence, Italy, July 2019. Association for Computational Linguisties. doi: 10. 10.18653/1/PP1--1166 URL httppp/:///.laatooooooooooo////p
Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A. Smith. Annotators with attitudes: How annotator beliefs and identities bias toxic language detection. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pp. 5884-5906 Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10. 10.18855//ll 1/t.22. naacl-main 1. URL wtt.tsss//acclantooog.oocoo
13

---

Published as a conference paper at ICLR 2024
Elvis Saravia, Hsien- Asien-Chi Toby Liu, Yen- Hao Huang, Junlin Wu, and Yi-Shin Chen. CARER: Contex- tualized affect representations for emotion recognition. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3687-3697, Brussels, Belgium, October- November 2.18. Association for Computational Linguistics. doi: 10. 10.18653/11/D188144 URL
https://aclanchology) ott.:///aaaaaaooooooorr///1---10
Melanic Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying language models' sensitivity to spurious features in prompt design How learned to start worrying about prompt formatting. In Proceedings of the International Conference on Learning Representations (ICLR), 2024. URL https://openreview. net/foorm?iidRRuullyyl
Emily Sheng and David Uthus. Investigating societal biases poetry composition system, 2020.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631-1642, Seattle, Washington, USA, October 2013. Association for Computational Linguistics. URL https::/aalaanhooooyy.oooo 0ttp:/ooyy.rg/1111112200
Taylor Sorensen, Joshua Robinson, Christopher Rytting, Alexander Shaw. Kyle Rogers, Alexia Delorey, Mahmoud Khalil, Nancy Fulda, and David Wingate. An information-theonm -hhorettc approach to prompt engineering without ground truth labels. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 519-862, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v 1 acl.oooooo.o. URL https: http:///aalaathhoogggyo wrg. orgg/022.acl.. acl-long.666 60
Jiuding Sun, Chantal Shaib, and Byron Wallace. Evaluating the zero-shot robustness of instruction- tuned language models, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprini arXiv:2302.13977 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint w3Xv:2307.09288 2023b.
Svitlana Volkova, Theresa Wilson, and David Yarowsky. Exploring demographic language variations to improve multilingual sentiment analysis social media. Proceedings of the 2013 Conference Empirical Methods in Natural Language Processing, pp. 1815-18211 Seattle, Washington, USA, October 2013. Association for Computational Linguistics. URL https:://clanttho.oog org/D13-1187
Johannes von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent, 2023.
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://gii thub. comp/kinggoflllz/mmesh transformer-jjaa May 2021.
Xinyi Wang, Wanrong Zhu, Michael Saxon, Mark Steyvers, and William Yang Wang. Large language models are implicitly topic models: Explaining and finding good demonstrations for in-context learning, 2023.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022. URL httpss///ppeneeeieeeeeeerrrrrroo gEZrGCoZdqR)
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. In International Conference on Learning Representations, .ntt./forrummdd-ddddFhhiii 2022. URL / http://s//ogeeeeeeeeeee/////errr
14

---

Published as a conference paper at ICLR 2024
Kenji Yamada and Kevin Knight. A syntax-based statistical translation model. In Proceedings of the 39th Annual Meeting of the Association for Computational Linguistics, pp. 523-530, 2001.
Yi Yang and Jacob Eisenstein. Overcoming language variation in sentiment analysis with social attention. Transactions of the Association for Computational Linguistics, 5:295-301, 2017. doi: 10.1 N0.11622talLa_0000 URL httptttss///ccaatttoo http:////ccccaatoooooooorrr////0
Kyra Yee, Yann Dauphin, and Michael Auli. Simple and effective noisy channel modeling for neural machine translation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp- (696-5701, 2019.
Yogatama, C Dyer, W Ling, and P Blunsom. Generative and discriminative text classification with recurrent neural networks. In T'hirty-ourthh International Conference on Machine Learning (ICML 2017). International Machine Learning Society, 2017.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205. 01068, 2022.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Gamett (ods.). Advances in Neural Information Processing Systems, volume 28, Curran Associates, Inc., 2015a. URL httpss://proceedding: wwwp:/eceeeeennn.eeeeeeer..e.s.. wttpppe.ccpppp..pppppppcpp 130c1511775$58a6bbee66666----9-2
httprripp.ccpppppee..ee/////rrrp https///p/e/pppe////////ss .pdf
Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In NIPS, 2015b.
Yufeng Zhang, Fengzhuo Zhang, Zhuoran Yang, and Zhaoran Wang. What and how does in-context learning learn? bayesian model averaging, parameterization, and generalization, 2023.
Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models, 2021.
Chunting Zhou, Xuezhe Ma, Paul Michel, and Graham Neubig. Examining and combating spurious features under distribution shift. In International Conference on Machine Learning, pp. 12857- 12867. PMLR, 2021.
A CONNECTION TO SURFACE FORM COMPETITION
Discriminative classification from language models have been shown to suffer from "surface form competition" where multiple surface forms of the label y, may compete for probability mass. To address this issue, Holtzman et al. [2021); Zhao et al. (202)) proposed calibrating the probability by using point-wise mutual information (PMI) between the text and the label the scoring function, p(x.v.)
which is given as, ý = arg max,, p())p While these works have simplified PMI as p( (o)lx)//p(())
an alternative way simplify it is p(x ((x)))) / /p(xx) which is the same as the generative classification setup as described above.
B ADDITIONAL EXPERIMENTAL DETAILS
Discriminative Baselines Variations For each discriminative zero-shot baseline, we consider three variations (see Table 6): (1) DISC-NONE does not condition the input text contextual variables, (2) DISC-CONTEXT conditions the input text on the contextual variables using simple format, (3) DISC-INSTRUCT conditions the input text the contextual variables as well as the labels that the model is expected to predict. In the main paper, we present results with DISC-NONE, which performs best out of these variations.
Table Jsummarizess all the datasets we use. Table 5 summarizes the hand written templates we start with (and later paraphrase to construct label descriptions).
15

---

Published as a conference paper at ICLR 2024
Dataset
Task (Domain)
#Classes
SST2
Socher et al..
2013
Sentiment Classification (movie)
SST5 Socher et al.
2013
Sentiment Classification (movie)
Yelp2 Zhang et al.
2015a
Sentiment Classification (Yelp)
Yclp5 Zhang al.
2015a
Sentiment Classification (Yelp)
5
2020 Poem Sentmmet (PS) (Sheng & Uthus
Sentiment Classification (Poctry)
Financial Phrascbank (FP) (Malo ct al.
2014
Sentiment Classification (Economic News)
MR (Pang & Lec 2005
Sentiment Classification (Rotten Tomatoes)
2
CR (Hu & Liu. (0044
Sentiment Classification (Customer Reviews)
2
Tweet3 (Rosenthal al. 2017
Sentiment Classification (Twitter)
AGNews (Zhang et al. 2015b
Topic Classification (News)
DBpedia
Topic Classification (Wikipedia)
14
ciite_speech& (de Gibert al.
2018
Hate Speech (Stormfront)
Ethos (4 subsets) (Molaa ct al. 2022
Hate Speech by Subject (various social media)
Emotion (Saravia et al. 2018
Emotion Recognition (Twitter)
Potato Prolinc (Per & Jurgens
2023
Politeness Classification (Email)
Talk Up (Njoo al. 2023
Empowerment prediction (Reddit)
Table 4: Datasets used for the experiments
Task
Label Description
Sentiment
"This [DOMAIN] leans [POLARITY]: "; DOMAINE movie review, Rol- ten'Tomatoes review, tweet, customer review, Yelp review, poem verse, financial excerpt), POLARITYE ( positive, positive. neutral, negative, very negative). We use "positive" and "negative" as POLARITY hinary sentiment classification.
Hate speech
"This [DOMAIN] uses [LABEL] language: "; DOMAIN Stormfront post), LABELE (hateful, innocuous).
Ethos
"This [DOMAIN] contains hate-speech about [SUBJECT]: "; DOMAINE (text, social media post), SUBJECTe (OECCT=(Oom)) (U national origin, religion, race, sexual orientation). ). This is binary classification task where "something" serves as the negative class for every other subject.
Topic
"The topic of this [DOMAIN] revolves around [TOPIC]: "; DOMAINe(tex.
news excerpt], TOPICE world, sports, business, technology] for AGNews, TOPICE TOPICE|compan ccompany, educational institution, artist, athlete, office holder, means of transportation, building, natural place, village, animal, plant, album, film, written work]
Politeness
"According to JAGE] years old person with a [EDUCATION], this email snippet is impolite:"; AGEE set of integers, EDUCATIONE (High school degree, college degree, Both are provided in the test example.
Emotion
"This [DOMAIN] emotes [EMOTION]: "; DOMAIN=(txxt, tweet), EMO- TION=(sadness, love, anger, joy, fear, surprisc)
Empowerment
"This Reddit comment written by [AUTHOR] empowers and uplifts the ad- dressed [ADDRESSEE]: AUTHOR= woman), ADDRESSEE=(mann
woman
Table 5: Label description starter templates we hand write. DOMAIN="text" represents missing domain information (which is one of our ablation). We generate their variations by asking ChatGPT: "Write 11 paraphrases of this sentence as a Python list." The full list is provided in the supplementary material.
C ADDITIONAL RESULTS
We provide averaged macro-Fl for each the models and each zero-shot method we consider in Figures
10
16

---

Published as a conference paper at ICLR 2024
Name DISC-NON NONE DISC-CONTEXT DISC-INSTRUCT
Format
[INPUT][LABBER DESCRIPTION!"
"This a [DOMAIN]. [INPUT] [LABEL DESCRIPTIONT"
"Is this [DOMAIN] [LIST OF LABEL NAMESJ? [INPUT] [LABEL DESCRIP-
TION]"
Table 6: Different variations of DISC models. The results reported in the main paper are from DISC-NONE as it performed the best overall. all three settings the label descriptions always contain the contextual information. The results for all three approaches can be found in Figures and
Figure 3: Full results for domain aware classification for our zero-shot DISC-NONE setup without calibration. The x-axis shows number of label descriptions per label and the y-axis indicates the average accuracy across the tasks except (Politeness and Empowerment). We conduct thorough analysis of this setup: removing domain information from the descriptions, different aggregation strategies as well evaluating on different model sizes and families.
17

---

Published as a conference paper at ICLR 2024
Figure 4: Full results for domain aware classification for our zero-shot DISC-CONTEXI setup without calibration. The x-axis shows number of label descriptions per label and the y-axis indicates the average accuracy across the tasks except (Politeness and Empowerment). We conduct thorough analysis of this setup as discussed in (section 4) removing domain information from the descriptions, different aggregation strategies as well as evaluating on different model sizes and families.
18

---

Published as a conference paper at ICLR 2024
110
Figure 5: Full results for domain aware classification for zero-shot DISC-INSTRUCT setup without calibration. The x-axis shows number of label descriptions per label and the y-axis indicates the average accuracy across the tasks except (Politeness and Empowerment). We conduct thorough analysis of this setup as discussed in (section 4) removing domain information from the descriptions, different aggregation strategies as well as evaluating on different model sizes and families.
19

---

Published as a conference paper at ICLR 2024
Figure 6: Full results for domain aware classification for our zero-shot DISC-NONE setup with PMI based calibration. The x-axis shows number of label descriptions per label and the y-axis indicates the average accuracy across the tasks except (Politeness and Empowerment). We conduct thorough analysis of this setup as discussed in (section 4) removing domain information from the descriptions, different aggregation strategies as well as evaluating on different model sizes and families.
20

---

Published as a conference paper at ICLR 2024
B-baloytsta
Figure 7: Full results for domain aware classification for our zero-shot DISC-CONTEXT setup with PMI based calibration. The x-axis shows number of label descriptions per label and the y-axis indicates the average accuracy across all the tasks except (Politeness and Empowerment), We conduct thorough analysis of this setup discussed in (section 4): removing domain information from the descriptions, different aggregation strategies as well as evaluating on different model sizes and families.
21

---

Published as a conference paper at ICLR 2024
Figure 8: Full results for domain aware classification for our zero-shot DISC-INSTRUCT setup with PMI based calibration. The x-axis shows number of label descriptions per label and the y-axis indicates the average accuracy across all the tasks except (Politeness and Empowerment). We conduct thorough analysis of this setup discussed in (section 4): removing domain information from the descriptions, different aggregation strategies as well as evaluating on different model sizes and families.
22

---

Published as a conference paper at ICLR 2024
0.0
0.55
013
055
A.0 Towakinoss
(a) GPT-2 Small
(b) GPT-2 Medium
(c) GPT-2 Large
0208
0335
AfC altrov Take AAN--TTaAAAnse--r
0.05
031
033
(d) GPT-2 XI.
(c) OPT 1.3B
(f) OPT 2.7B
338
6.4
(g) Pythia 1.4B
(h) Pythia 2.8B
Figure Full results for author and personalzed empowerment prediction with our proposed setup. The x-axis shows number of label descriptions per label and the y-axis indicates the average Fl-score. The four plots indicate 4 settings described Table
23

---

Published as a conference paper at ICLR 2024
0.7
677
HiluAuuor 200.5 The
HARA AOACE
(a) GPT-2 Small
(b) GPT-2 Medium
(c) GPT-2 Large
0.50
042
0.6
0358
(d) GPT-2 XI.
(c) OPT 1.3B
(f) OPT 2.7B
3.00
3.15
2.7
0756
0.6
070
0.20
(g) Pythia 1.4B
(h) Pythia 2.8B
Figure 10: Full results for reader-pereee padder-personalized politeness prediction with our proposed setup. The x-axis shows number of label descriptions per label and the y-axis indicates the average accuracy. The four plots indicate settings described Table
24


Reviews:
[{"review_id": "KQ35vgyVRp", "reviewer": "Reviewer_vWvw", "summary": "This paper introduces a generative prompting framework for zero-shot text classification. It proposes to make text classification based on the LMs' likelihood of input text conditioned on the description of each label, where the label description comes from human annotation and ChatGPT. Experiments show this proposed model achieves solid zero-shot text classification performance over baselines.", "questions": "Do you think the proposed framework can be used to amplify few-shot classification systems? I think it may make better sense in assisting a few-shot classification system since adding a few more training samples may not be as effective as providing these label descriptions given the same annotation efforts.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper is well-written and easy to follow.\n- The empirical analysis is comprehensive and solid regarding datasets and backbone models.\n- The proposed method is simple yet effective, providing a solid and lightweight framework to solve zero-shot text classification problems.", "weaknesses": "The proposed method, presentation, and empirical analysis are well self-explained. I don't have much concern about it. \n\nMy main concern is whether the problem of \"zero-shot text classification with some human/ChatGPT labeled data\" is meaningful. If one would like to do some annotation, either with human annotators or ChatGPT, why not directly annotate some labeled samples for few-shot text classification? Thus, in my opinion, to show that this problem setup and the proposed framework are meaningful, the authors need to somehow show that this zero-shot framework is more effective than a few-shot text classification framework given the same amount of annotation effort.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "49oT6nbjhq", "reviewer": "Reviewer_uHMN", "summary": "This study suggests employing multivariate generative classification over discriminative approaches for zero-shot text classification tasks utilizing in-context learning. \nBuilding on Min et al. (2022), the paper demonstrates that through multiple iterations with varied contexts elucidating label information, generative classification coupled with in-context learning emerges as a more rational and stable solution.\nFurthermore, users can enhance the context with additional information (e.g., the gender of a writer) to more effectively clarify the label space. \nIt is evidenced that this supplementary context can significantly improve the performance of in-context learning in zero-shot classification tasks.", "questions": "- What potential impacts could arise from the combination of the proposed method with few-shot learning scenarios? This consideration is crucial, especially given that while generative zero-shot classification yields reasonable results, it does not yet match the performance of few-shot-based methods, like ICL (DC) as depicted in Table 2.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The authors successfully derived the final form of the probability for generative text classification (i.e., $\\sum_{z_i} p(\\mathbf{x}|z_i)$), based on a series of reasonable assumptions.\n- The paper outlines a well-considered experimental setup that demonstrates the efficacy of the proposed method in specific applications like domain-aware classification and personalized classification.", "weaknesses": "- The method proposed is conceptually similar to Min et al.'s noisy channel method, with the primary distinction being the use of multiple \"label descriptions\". This similarity raises questions regarding the novelty of the proposed approach.\n- The assumptions outlined in Section 2.3 may be overly simplistic, potentially detracting from the final performance of the proposed method. The paper would gain interest if the authors could suggest reasonable approximations for the omitted probabilities, namely i.e., $p(z|y_i, u,v,\\dots)$ and $p(y_i|u,v,\\dots)$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9c7nv51Nxh", "reviewer": "Reviewer_BREF", "summary": "This work proposes GEN-Z, a generative prompting framework for zero-shot text classification. Specifically, GEN-Z measures the LM likelihood of input text, conditioned on natural language descriptions of labels, which can be extended to multivariate forms. This work conduct experiments on multiple classification tasks. Experimental results show that the proposed framework can achieve better performance compared with the baseline approaches.", "questions": "1. For LMs, the context can influence the language model probability significantly. Combining multiple label descriptions may result in changed language model probability. So how is multiple label descriptions utilized in this work?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed framework is interesting and seems novel. Utilizing label description in the bayesian form can help to improve the overall classification performance as shown in the experiments in the paper.\n\n2. This work conducts extensive experiments on a variety of datasets, including domain-aware classification and personalized classification. This is very encouraged in NLP research.", "weaknesses": "1. Though this work argues that utilizing label descriptions in a bayesian form benefits the performance, it relies heavily on the quality of the label description, e.g., how clear the context can present the task, domain the label. However, in text classification, sometimes the label is not natural language, but a less literally meaningful symbol. For such labels, it is not likely to encourage the model to generate text $x$ given $y$. Then it requires human beings to provide very good summaries or descriptions for such labels, which is sometimes challenging. Such a problem limits the application of the proposed GEN-Z framework. Instead, conventional in-context learning can provide examples and have the model to learn from simpler label descriptions and demonstrations, which is more flexible.\n\n2. Since label descriptions directly decides how the model generates $x$, the quality of $z_i$ plays an important role on the performance. It is encouraged to explore how the performance is influenced if descriptions of different quality are given. For example, if the label can already express the meaning of the label, then descriptions may not be required. When the label meaning is more abstract, detailed descriptions may be better than abstract descriptions. Such influence is encouraged to be explored.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3KvSNXH86n", "reviewer": "Reviewer_N7Hu", "summary": "This paper proposes a generative prompting framework for zero-shot text classification. Specifically, the proposed model takes on the text classification task by measuring the LM likelihood of the input text, conditioned on natural language descriptions of labels.GEN-Z leverages LM likelihood of generating the input text based on various label descriptions that reflect context, enabling more robust\npredictions than discriminative approaches. Experiments show that GEN-Z consistently improves classification performance\nover zero-shot baselines and performs on par with strong few-shot baselines", "questions": "Do you add the label description to other baseline models, which are generated from ChatGPT? If not, the comparison may not be fair because you added more external knowledge. \n\nCan the generative classifier be used to other suan as multimodel, or image data?(CLIP)", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) This generative in-context learning is natural due to the LLM being based on a generative model. This paper propose two novel module based on baseline models, including MULTIVARIATE GENERATIVE CLASSIFICATION and CONTEXTUALIZED LABEL DESCRIPTIONS. \n\n2) Compared with other in-context learning methods, this method can improve robustness effectively.\n\n3) The paper is well written, and the figure is helpful for understanding.\n\n4) The analysis of the Baysian equation is novel, simple, and clear.", "weaknesses": "This paper may take incremental work, which adds two new modules based on the generative classifier in [1].\nIn my opinion, the work doesn't show any new insight, maybe not enough for ICLR but more suitable for other conference , such as ACL.\n\n[1] Min, Sewon, et al. \"Noisy channel language model prompting for few-shot text classification.\" arXiv preprint arXiv:2108.04106 (2021).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)