PIEClass: Weakly-Supervised Text Classification with Prompting and Noise-Robust Iterative Ensemble Training
Authors: Yunyi Zhang
Minhao Jiang
Yu Meng
Yu Zhang
Jiawei Han
Venue: EMNLP 2023 Main (2023)
Keywords: Text Classification
Weak Supervision
Pre-Trained Language Model
Prompt-Based Learning

Abstract: Weakly-supervised text classification trains a classifier using the label name of each target class as the only supervision, which largely reduces human annotation efforts. Most existing methods first use the label names as static keyword-based features to generate pseudo labels, which are then used for final classifier training. While reasonable, such a commonly adopted framework suffers from two limitations: (1) keywords can have different meanings in different contexts and some text may not have any keyword, so keyword matching can induce noisy and inadequate pseudo labels; (2) the errors made in the pseudo label generation stage will directly propagate to the classifier training stage without a chance of being corrected. In this paper, we propose a new method, PIEClass, consisting of two modules: (1) a pseudo label acquisition module that uses zero-shot prompting of pre-trained language models (PLM) to get pseudo labels based on contextualized text understanding beyond static keyword matching, and (2) a noise-robust iterative ensemble training module that iteratively trains classifiers and updates pseudo labels by utilizing two PLM fine-tuning methods that regularize each other. Extensive experiments show that PIEClass achieves overall better performance than existing strong baselines on seven benchmark datasets and even achieves similar performance to fully-supervised classifiers on sentiment classification tasks.

PIEClass: Weakly-Supervised Text Classification with Prompting and Noise-Robust Iterative Ensemble Training
Yunyi Zhang, Minhao Jiang, Yu Meng, Yu Zhang, Jiawei Han University of Illinois Urbana-Champaign, IL, USA [yzhan238, minhaoj2, yumeng5, yuz9, hanj janjeeill wiw.iiiiiiiim edu
Abstract
text classifiers in a fully-supervised manner that re- quires a substantial amount of training data (Zhang et al., 2015; Yang et al., 2016), which are generally costly to obtain. To eliminate the need for labeled training samples, weakly-supervised text classifica- tion settings (Meng et al., 2018, 2020; Wang et al., 2021) are proposed, which aim train text clas- sifiers using the label names of target classes the only supervision. Such settings are intriguing especially when obtaining high- high-quality labels is prohibitively expensive.
Weakly-supervised text classification trains classifier using the label name of each target class as the only supervision, which largely reduces human annotation efforts. Most exist- ing methods first use the label names as static keyword- -baee features to generate pseudo la- bels, which are then used for final classifier training. While reasonable, such a commonly adopted framework suffers from two limita- tions: (I) keywords can have different mean- ings in different contexts and some texts may not explicitly contain any keyword, so key- word matching can induce noisy and inade- quate pscudo labels; (2) the errors made in the pseudo label generation stage will directly propagate to the classifier training stage with- out chance being corrected. In this paper, we propose new method, PIEClass, consist- ing of two modules: (1) a pseudo label ac- quisition module that uses zero-shot prompt- ing of pre-traine pre-trained language models (PLM) to get pseudo labels based on contextualized text understanding beyond static keyword match- ing, and (2) noise-robust iterative ensemble training module that iteratively trains classi- fiers and updates pseudo labels by utilizing two PLM fine-tuning methods that regularize each other. Extensive experiments show that PIEClass achieves overall better performance than existing strong baselines on seven bench- mark datasets and even achieves similar perfor- mance to supe-veuee classifiers on senti- ment classification tasks.'
Recent advancements in large generative lan- guage models (LLMs) (c.g.. ChatGPT, GPT-4 (Ope- nAI, 2023)) make it a valid approach to directly prompt them in a zero-shot manner for text classifi- cation without labeled data. For example, people may provide restaurant review and ask an LLM "What is the sentiment of this document?", and the model will generate an answer according to its un- derstanding. However, there are certain limitations of this method for the reakry-superviied text clas- sification setting. First, directly prompting LLMs cannot utilize any domain-specific information hid- den in the unlabeled data, because it is intractable to funetune such large model and the prompts can hardly incorporate any corpus-level informa- tion, especially for corpora appearing in LLMs' pre-training data (e.g.. private domains). Second, deploying L.Mss is expensive, while many text classification applications require fast real-time in- ference (e.g., email and review classification).
Another line of studies tailored for weakly- supervised text classification aims to train a moderate-size classifier with a task-specific unla- beled corpus. Given the label names, these meth- ods first acquire class-indicative keywords using PLMs (Meng al., 2020; Wang al., 2021) corpus-level co-occurrence features (Zhang et al., 2021, 2022b). The keywords are then used as static features to generate pscudo-labeled docu- ments for fine-tuning the final classifier. De-
1 Introduction
Text classification is fundamental NLP task with a wide range of downstream applications, such as question answering (Rajpurkar et al., 2016), sen- timent analysis (Tang et al., 2015), and event de- tection (Zhang et al., 2022c). Earlier studies train
Code can be found at ht sttps://g/t wtt.hh.b.ccmmyhaa33////// PIEClass.

---

spite their promising performance, the aforemen- tioned
way, we can gradually expand the pseudo labels which can be used train better text classifiers. To avoid erroneous pseudo labels accunulated dur- ing the iterative process, we propose to utilize two PLM fine-tuning strategies, head token fine-tuning and prompt- based fine-tuning, as two complemen- tary views of the data: One captures the semantics of the entire sequence while the other interprets the contexts based on the prompts. We use the two views to regularize each other and further apply model ensemble to improve the noise robustness of the pscudo label expansion process.
methods may suffer from two major limitations. First, these methods are keyword-driven by using class-indicative keywords as static context-free features to generate pseudo labels with different forms of string matching. However, some texts may not contain such class- indicative keywords and keywords may have differ- ent meanings in different contexts, so using them as static features can lead to noisy and inadequate pseudo labels. Such an issue is more serious for abstract classes like sentiments that require under- standing rhetoric. For example, a food review "Ir is to die for!" contains the keyword "die" which itself is negative, but the entire review expresses a strong positive sentiment, and keyword-driven methods will likely struggle in these cases. Second, most existing methods are two-stage by conducting pseudo label acquisition and text classifier train- ing in two successive steps. Although different pseudo label acquisition methods are explored to improve their quality (e.g., masked language mod- eling (Meng et al., 2020), clustering of PLM em- beddings (Wang et al., 2021), or large textual entail- ment models trained with external data (Park and Lee, 2022)), there is still large performance gap between weakly-supervised and fully-supervised settings, because erroneous pseudo labels in the first stage will propagate to and harm the classifier training stage without a chance to be corrected.
To summarize, the contributions of this paper are as follows: (1) We propose to use the contextu- alization power of PLM prompting to get pseudo labels for the -uukely-upervised text classification task instead of static keyword-based -bsedd features. (2) We explore the prompting method of a discrimina- tive PL. on the text classification task and com- pare it with prompting methods for MLM. (3) We propose a noise-robust iterative ensemble training method. To deal with noisy pscudo labels, we uti- lize two PLM fine-tuning strategies that regular- ize each other and apply model ensemble to en- hance the pseudo label quality. (4) On seven bench- mark datasets, PIEClass overall performs better than strong baselines and even achieves similar performance to fully-supervised methods.
2 Problem Definition
The supely--upervised text classification task aims to train a text classifier using label names as the only supervision. Formally, given set of doe- uments D = (di,..., dn} and m target classes C {cI...., Cm] with their associated label names 1(c), our goal is train text classifier F that can classify a document into one of the classes. For example, we may classify collection of news articles using the label names "politics", "sports", and "technology". Notice that, there are previous studies using more than one topic-indicative key- word or a few labeled documents supervision, whereas here, we follow the extremely weak super- vision setting (Wang et al., 2021) and only use the sole surface name of each class as supervision.
To address the limitations of existing works, in this paper, we propose PIEClass: Prompting and Iterative Ensemble Training for Weakly- Supervised Text Classification. PIEClass consists of two modules. (1) Pseudo label acquisition via PLM prompting. By designing task-specific prompt, we can apply a moderate-size PLM to infer the class label of documents based the entire input sequence, which is thus contextualized and beyond static keyword features. In this work, besides the well-studied prompting method using PLMs pre-trained with the masked language mod- eling task (MLM) g., BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019)), we also explore a dif- ferent prompting method for a discriminative pre- trained language model, ELECTRA (Clark et al., 2020), and compare them in the experiments. (2) Noise-robust training with iterative ensemble. In
Methodology
To address the limitations of existing methods for reakly-supervised text classification, we introduce our method, PIEClass, in this section, which con- tains two major modules: (1) zero-shot prompting
each iteration, we train text classifiers using the current pseudo labels and then use the confident predictions to re-select the pseudo labels. In this

---

Prodictions
Prediction
Yacubulary
Prodictions
Prob. original: 0.8
Prob, of original: 0.2
Pre-Trainet Language Model
Pre- Trainrr Language Model
Pre Trained Language Model
same
Pre- Train-randd Language Model
4 [CLS] It to die for!
? It to die for was good,
is die fort was [MASK]
It is to die for was bad,
Input Text
Prompr
Armppr
Input
Prompr
Head Token Fine-Tuning
Prompt-Based Fine-Tuning (MUM)
Prompt-Based Fine-Tuning (ELECTRA)
Figure I: Examples of different fine-tuning strategies on the text classification task. (left) Head token fine-tuning randomly initializes linear classification head and directly predicts class distribution using the [CLS] token, which needs substantial amount of training data. (middle) Prompt-based fine-tuning for MLM-based PLM converts the document into the masked token prediction problem by reusing the pre-trained MLM head. (right) Prompt-based fine-tuning for L.ECCRA-stye PLM converts documents into the replaced token detection problem by reusing the pre-trained discriminative head. Given a document, one input sequence is constructed for each label name.
for pseudo label acquisition and (2) noise-robust training with iterative ensemble. Figure 2 shows overview of PIEClass.
shows examples of standard head token fine-tuning and the popular prompting method for MLM. Besides utilizing the MLM-based prompting method, in this work, we propose to exploit a dis- criminative PLM, ELECTRA (Clark et al., 2020), for zero-shot prompting. During pre-training, ELECTRA uses an auxiliary model to generate training signals and trains the main model to de- noise it. More specifically, a small Transformer model called "generator" is trained with masked language modeling to replace the randomly masked tokens of the input text, and then the main Trans- former model called "discriminator" trained predict whether each token the corrupted exam- ple is original or replaced by the generator (Clark et al., 2020).
3.1 Zero-Shot Prompting for Pseudo Label Acquisition
Most existing weakly-supervised text classification methods use set of static class indicattde key- words to assign pscudo labels to unlabeled docu- ments based on either direct string matching (Meng al., 2018) static class embeddings (Wang al.. 2021). However, keywords can only pro- vide limited supervision with low coverage, given that most of the documents do not contain any class- indicative keywords. Also, a document containing keywords does not necessarily indicate that it be- longs to the corresponding class because the key- words can mean differently in different contexts. Such issues are more serious for abstract classes that involve more rhetoric, such sentiment clas- sification. For example, a food review "It is to die for!" does not have any single keyword indicating the positive sentiment and even contains the word "die" " that seems negative, but we can still infer its strong positive sentiment based on our contextual- ized text understanding beyond static keywords.
Recent studies have shown the potential of ELECTRA in prompt-based methods (Xia et al., 2022; Yao et al., 2022: Li et al., 2022). Figure I (right) shows example. To generalize the us- age of ELECTRA-based prompting to weakly- supervised text classification, we can fill in a tem- plate 7ELECTRA with a document and one of the label names I(c). The template is designed in a way that the correct label name should be the "original" token of this constructed input while the wrong ones are "replaced". Take the sentiment classifica- tion task as example. If we want to classify whether restaurant review expresses posi- tive or negative sentiment given their label names "good" and "bad", we can construct the following two input sequences to ELECTRA,
To tackle the problem of existing methods and acquire pseudo labels beyond context-free keyword features, we propose to apply zero-shot prompting of PLMs. The prompt- based method aims to close the gap between the pre-training task of PLM and its downstream applications, pre-trained but not fine-tuned PLM with prompts get high-quality pseudo labels for the text classi- fication task. Also, prompting the PLM guides it to understand the entire context, and thus its predic- tions are contextualized. Figure 1 (left and middle)
can directly use
7 ELECTRA (d, good) = d It was good.
T ELECTRA bad) d It was bad.
The constructed inputs are individually fed into a pre-trained ELECTRA discriminator and its dis-

---

Head Token Fine-Tuning
Prompt- Based Ten-TTning
Protiae sestinett
0 (orignall
Two fine tuning strategies pre trained language model
Pre: Pre-Triined Language Model
Pre-Trained Language Model
[CLS] it to die for!
is die fort good,
Input
Initial Pseudo Labels p
Sampling
Prompt-Based Fine-Tuning
Updated Pseudo Labels p
Intersection
Zero-Shot
Head Token Fine-Tuning
Prompting
Sampling
Prompt- Prompt-Based
Fine-Tuning
Unlabeled
Corpus
Use updsted preudo lsbels to repeat the process
(1) Zero-Shot Prompting Pseudo Label Acquisition
(2) Noise-Robust Training with Iterative Ensemble
Figure 2: Overview of the PIEClass framework.
criminative classification head f to get the proba- bility of being original for each label name, p(((c)ld) Sigmoid( (f (+))),
wrongly labeled samples wrong with high confi- dence again. Therefore, we strictly follow the standard self-training method, the noise will stay and accumulate in the pseudo label pool. To tackle such challenge, we propose an iterative ensem- ble training method with two types of ensemble ensure the quality of pseudo labels. First, we utilize two PLM fine-tuning methods, head token and prompt-based fine-tuning, to train classifiers individually in each iteration. Here, the head to- ken fine-tuning behaves like a sequence-level view of documents by capturing the information of the entire input document, while prompt- based fine- tuning serves a token-level view by focusing more the context surrounding the label name (or masked token if using MLM) in the prompt. The two views can complement each other to better ex- ploit the power of PLMs. Second, since the prompt- based method converts the downstream task into the same form as the pre-training task and reuses the pre-trained classification head, it only requires small amount of data to achieve competitive per- formance with head token fine-tuning, This al- lows to further apply model ensemble by fine- tuning multiple individual prompt-based classifiers to further improve the noise-robust (Laine and Aila, 2017: Meng et al., 2021). Finally, in each iteration, we freshly initialize the classifiers with pre-trained weights, re-select all the pscudo labels, and only keep the top predictions agreed upon by all the classifiers to ensure the quality. Our idea shares similar spirit co-training (Blum and Mitchell, 1998). The major difference is that standard co- training learns from initial clean data and utilizes two data views to progressively label unlabeled
(1)
where (c()) is the contextualized representation of the label name (()) in this context. The confidence of document d belonging to a class is the normal- ized probability,
p(((c)ld)
p(cld) Ee'e p(((c']dd)"
(2)
After getting the predictions for all the documents in D. take the top t percentage of the docu- ments with the highest confidence scores as our initial pseudo label pool po.
3.2 Noise-Robust Training with Iterative Ensemble
With the initial pscudo labels, existing methods di- rectly fine-tune (using the head token) text classi- fier with such labels to get the final classifier. How- ever, since the pseudo labels are noisy with typical noise ratios ranging from 15% to 50% (Mekala al., 2022), the performance of the final classifier is limited by the quality of pscudo labels, lead- ing to a large performance gap between weakly- supervised and fully-supervised settings. There- fore, inspired by the self-training method for semi- supervised learning, we propose to iteratively train a text classifier and use its confident predictions to find more high-quality pseudo labels, which can help to train even better classifier.
However, unlike semi-supervised settings where the initial labels are perfect, here we only have noisy pseudo labels po from the last step. When we train a text classifier with noisy data as su- pervision, the classifier will likely predict those

---

data, whereas our method does have access to annotated training data but instead uses two data views as regularization along with model ensem- ble to improve model's nobse-robussness trained on pseudo-labeled data.
Algorithm 1: PIEClass
Input: A corpus 2; a set of classes C and their label names I(e). c C: a pre-trained language model E; a template T for prompting.
Output: A text classifier F for classes C.
Zero-Shot Prompting for Pseudo Label Acquisition;
Specifically, for iteration i, we first use the head token to fine-tune text classifier, Fj D - C. using the current pseudo labels pi-l in fully- supervised way. After training, we the classi- fier make a prediction on each document to get (dy, (dd)) and a confidence score efō(dz) which is the normalized probability of prediction (j)))) Then, we will rank the predictions based on their confidence scores and select the top 4 percentage of them whose confidence scores are greater than a threshold p as candidate pseudo labels Po- The threshold tę is linearly increasing with iteration numbers, = s, where hyperparameter. We use an increasing threshold because if we keep the threshold constant, the pseudo samples in the last iteration will be predicted confidently again, making the pseudo samples almost the same in the iterative process and the classifiers overfit more to the limited number of pseudo samples ?.
for d E D do
for E do
T(d, ((c]) + Construct input with
template;
p(((c]dd) Prompt with Eq. (I);
6 p(cld) + Eq. (2);
po top t percentage of predictions:
// Noise-Robust Training with Iterative Ensemble;
for <- 1 do
Aa + Head token fine-tuning using pi-1;
Pd Select top 4, predictions by Fe::
S + Randomly sample subsets of Po:
for S E S do
F Prompt-based fine-tuning using Ss:
15
P <- Select top t. percentage by F)::
p Bq. (4);
F 4- Head token fine-tuning using PT;
Return F;
Table I: Datasets overview.
Dataset
Classiftcation Type
Docs
Classes
AGNews
News Topic
120,000
Because Pjj can be noisy, we then utilize prompt- based fine-tuning
20News
Topic
17,871
NYT-Toples
News Topic
31,997
second view improve the quality of pseudo labels. We randomly sample subsets of P. Sg, each of size q [Pol. q < (0, and fine-tune r classifiers Fj. l E {1....,r}. using prompt-based fine-tuning. With a small sampling ratio q, the noisy labels are unlikely to be sampled repeatedly into different subsets, so this sampling process will further improve the noise robustness of model ensemble. To fine-tune ELECTRA-style PLMs using prompts, each data sample d will have (C] individual input sequences (TEEECTRA (d, ((c)) and the target class (d)) should be predicted as "original" while all the oth- ers "replaced"' The model trained with binary cross entropy loss
NYT-Fine
News Topic
13,081
26
Yelp
Business Review Sentiment
38,000
IMDB
Movie Reviow Seatiment
50,000
1)
Amazon
Product Review Sentiment
3,600,000
P
(4)
A=
The intersection operation can be interpreted as follows: a document and its assigned class belong to p only when it is consistently predicted as the same class and its confidence is ranked top 1, by all the classifiers Fj- Therefore, to include only those most confident ones into the pseudo label pool alleviate the error accumula- tion problem. The less confident predictions of the current iteration will only be left out for the current iteration, but will be re-examined in later iterations and added to pseudo labels if it is qualified.
ensure
CELECTRAA
=- Σ (oo))()(())d)++
dEg
Σ (
log (1 (1- (l-rl)ld))
Finally, we will repeat this iterative process by T full iterations get the last pseudo label pool PT. It will then be used for head token fine-tuning of the classifier iteration T which will be the final classifier of PIEClass. Algorithm 1 summarizes the entire framework.
After training, we follow the same process as the classifier F to select the top tú percentage of most confident predictions by each classifier Ft as can- didate pseudo labels Pi. Finally, we take the in- tersection of all the candidate pseudo labels as the final pseudo label pool for this iteration,
Experiments
4.1 Experiment Setup
"See Appx C.4 for some empirical results.

---

Table 2: Performance of all compared methods measured by Mooro-F1/Macro-FL with the best score boldfaced and the second best score underlined. We re-run ClassKG with its official implementation using only the label names for a fair comparison. Other baseline results come from (Meng et al., 2020) and (Wang et al., 2021) with
missing values marked as 'The results are influenced by RoBERTa's tokenizer.
Methods
AGNews
20News
NYT-Topies
NYT-Fine
Yelp
IMDB
Amazon
WeSTClass
0.823/08811
0.713/0.699
0.68300570
0.739/0.651
0.816/0.816
0.774/-
0.753/
ConWea
0.746/0.742
0,757/0.733
0,817/0.715
0.762/0.66 698
0.714/0.112
-/-
LOTClass
0.869/0.868
0.738/0.725
0.671/0.436
0.150/0.202
0.878/0.877
0.865/-
0,916/-
XClass
0.857/0.857
0.786/0.778
0.790/0.686
0.857/0.674
0,900/0.900
-/-
ClassKG
0.881/08881
0.811/0.820
0.721/0.658
7.88900705
0.918/0.918
0.888/0.888
0.926/-
RoBERTa (0-shot)
0,58 1/0.52
0,507/0.445
0,544/0.382
-f-1
0.812/0.808
0,784/0.780
0.788/0.783
ELECTRA (0-shot)
0.810/0.806
0.558/0.529
0,739/1.613
0.765/0.619
0.820/0.820
0.803/0.802
0.802/0.801
PIEClass
ELECTRA+BERT
0.884/0.884
0.789/0.791
0,807/0.710
0.898/0.732
0.919/0.919
0.905/0.905
0,858/0.858
ROBERTn-ROBERTH
0.895/0.895
0.755/0. 760*
0.760/0.66 694
.f+$
0.920/0.920
0,906/0.900
0.912/0.912
ELECTRA+ELECTRA
0.884/0.884
0.816/0.817
0.832/0.763
0.910/0.776
0,957/0957
0.931/0.931
0.937/0.937
Fully-Supervised
0.940/0.940
0.965/0.990 964
0,943/0.899
0.980/1.966
0.957/0.957
0.945/-
0.972/-
Datasets We use 7 publicly available benchmark datasets for the weakly-supervised text classifica- tion task. Four for news topic classification: AG- News (Zhang et al., 2015), 20News (Lang, 1995), and NYT-Topies (imbalanced) and NYT-FFine (im- balanced and fine-grained) (Sandhaus, 2008); three for sentiment classification: Yelp (Zhang et al., 2015), IMDB (Maas et al., 2011), and Ama- zon (McAuley and Leskovec, 2013). Table I shows the data statistics, and Table 4 in the Appendix shows the label names and prompt used for each dataset. We follow previous works to Miero- F1/Macro-F1 as the evaluation metrics.
Run Time on 20News
XClass
0.7hr
LOTClass
1hr
PIEClass
3hr
ClassKG
30hr
20
30
Figure 3: Run time (in hours) on 20News. ClassKG takes much longer time than other methods.
ELECTRA+ELECTRA uses ELECTRA backbone models for the entire framework. The implementation details of PIEClass are in Appx B.3.
4.2 Experimental Results
Compared Methods We compare the following methods on the weakly-supervised text classifica- tion task: seed-driven methods WeSTClass (Meng et al., 2018 8) and Con Wea (Mekala and Shang, 2020), which take at least three keywords for each class input; LOTClass (Meng et al., 2020), XClass (Wang al., 2021), and ClassKG (Zhang et al., 2021) that only take label names as supervi- sion; two zero-shot prompting baselines RoBERTa (0-shot) and ELECTRA (0-shot); and a Fully- Supervised BERT baseline as a reference. See more details of compared methods in Appx B.2. We include three versions of PIEClass with differ- ent combinations of backbone PLMs:
Table shows the evaluation results of all methods. PIEClass overall achieves better performance than the compared baselines. It even achieves similar results to the fully-supervised baseline on Yelp and IMDB. We can observe that: (1) ELECTRA+BERT model already outperforms most of the baselines that also use BERT to fine-tune their final clas- sifiers, which shows the effectiveness of pro- posed method. (2) ClassKG as an iterative method the strongest keyword-driven baseline and even achieves better results on 20News than PIEClass. However, it takes a drastically longer time to run. Figure 3 shows the run time on 20News. ClassKG takes more than 30 hours while PIEClass takes only 3 hours to achieve similar results. (3) ELEC- TRA (0-shot) already achieves comparable results to some simple baselines, confirming our idea that using contextualized text understanding can lead to high-quality pseudo labels. Although RoBERTa (0-shot) does not perform well on AGNews, af- ter the iterative classifier training process, the full
ELECTRA+BERT uses ELECTRA for prompt- ing and BERT for head token fine-tuning for fair comparison with baselines using BERT for final classifier training.
RoBERTaaRRBERR' uses RoBERTa as back- bone models for the entire framework to compare with baselines using only MLM-based PLM.

---

Table 3: Performance of PIEClass (ELECTRA+BLECTRA) and its ablations measured by Micro-F1/Macroy F1/Macro-MMM.
Methods
AGNews
20News
NYT-Topics
NYT-Fine
Yelp
IMDB
Amazon
Two-Stage
0.847/0.847
0.739/0.733
0.776/0.664
0.838/0.678
0.913/0.913
0.870/0.870
0.836/0.835
Single-View View ST
0.871/08871
7.7
0.757/0.666 668
0.853/0.695
0,912/0,912
0.846/0.846
0.892/0.892
Co-Training
0.877/0.877
0.795/0.791
0.818/0.715
0.877/0.744
0,948/0.948
0.925/0.925
0,930/0,930
PIEClass
0.884/0.884
0.816/0.817
0.832/0.763
0.910/0.776
0.957/0.957
0.931/0.931
0.937/0.937
model achieves the best performance, demonstrat- ing the effectiveness of the iterative process of PIEClass. (4) ELECTRA overall performs better than RoBERTa, especially on the sentiment classi- fication task. Also, RoBERTa's performance can be affected by its tokenizer: in 20News, the label name "religion" is separated into two tokens, so we have to use a sub-optimal label name; half of NYT-Finees label names are tokenized into multi- ple pieces, so we do not report the performance.
same hyperparameters as described Appx B.3. Table 3 shows the performance of PIEClass and its ablations on seven datasets. We can observe that: (1) our full model PIEClass consistently out- performs all of its ablations, showing the effective- ness of each ablated component. (2) By removing the iterative pseudo label expansion process, the Two-Stage model performs worse than PIEClass, meaning that the erroneous pseudo labels in the first stage will affect the final classifier training if not corrected. However, the Two-Stage version already achieves comparable results to strong keyword- driven baselines, which shows the power of zero- shot PLM prompting on the text classification task. (3) The Single- View ST model performs similarly to the Two-Stage model and sometimes even worse. This proves that, with the noisy pseudo labels, the standard self-training strategy can cause the error accumulation problem and harm the classifier train- ing. (4) The Co- Co-Training model performs much better than the previous two ablations, meaning that utilizing
To explain why PIEClass can achieve similar performance the fully-supervised method, find that there are some errors in the ground truth labels which could affect the performance of fully- supervised model if used as training data. For example, the following review in Yelp labeled as negative but predicted as positive by PIEClass: " My husband had an omelette that was good. I had a BLT, little on the small side for $10, but bacon was great. Our server was awesome!". Because PIEClass only includes the most confident predic- tions as pseudo labels, it can ensure the quality of its training samples to make the correct prediction.
PLM fine-tuning methods as views of data can improve the pseudo label qual- ity. However, it still performs worse than PIEClass, showing that using two views to regularize each other can further improve the noise robustness.
4.3 Ablation Study
To study the effects of each proposed component of PIEClass, we further compare our full model with its three ablations:
To show the effectiveness of our pseudo la- bel generation and selection, we also compare PIEClass with vanilla few-shot classifiers on AG- News and IMDB. While PIEClass does not need any label, we find that around 500 1,000 labels are needed for the few-shot classifiers to achieve similar performance as PIEClass. More details are described in Appx C.I.
Two-Stage is a two-stage version of PIEClass which directly trains the final text classifier using the pseudo labels from zero-shot prompting. Single-View View ST (Self-Training) does not uti- lize prompt-based fine-tuning as a second view during the iterative process. It thus follows standard self-training method by using the con- fident predictions of the head token classifier as the updated pseudo labels for the next iteration. Co-Training uses the two views of data (i.c., two PLM fine-tuning strategies) to update the pseudo labels in turn with their confident predic- tions, while in PIEClass the two views used to regularize each other.
4.4 Study of the Iterative Process
To study the iterative process of PIEClass, we show the performance of PIEClass and its Single- View Self- Trainirain ablation when varying the number of full iterations from to 5 in Figure 4. From the figure, we can see that, although Single-View Self- Seainigg may perform better than PIEClass when the quantity of pseudo labels is small at the beginning, after five iterations, PIEClass consis-
All the compared methods are based on the ELEC- TRA+EL version of PIEClass with the

---

20Nsws
Yelp
Weakly-Superviied Text Classification Weakly-supervised text classification aims to train a classifier with very limited supervision. Earlier studies utilize distant supervision from knowledge bases such as Wikipedia to interpret the document-labei semantic relevance (Gabrilovich and Markovitch, 2007; Chang et al., 2008; Song and Roth, 2014). Some other supervision signals such as keywords (Agichtein and Gravano, 2000; Tao et al., 2018; Meng et al., 2018, 2020; Wang et al., 2021; Zhang et al., 2021) and heuristic rules (Ratner et al., 2016; Badene et al., 2019; Shu et al., 2020) are also explored to reduce the efforts of acquiring any labels or domain-specific data. Recently, the extremely
PiEOau
PlEClus
Seglo
Siigl
0.75
1.50
0.70
0.55
bouises
Narbes ittation
Figure Performance of PIEClass and Single- View ST
across varying numbers of full iterations.
Yelp (PIEChee)
Yoln (Singl=Vinn ST)
47 1g
900 53
9ys qo
22
Swcations
Figure 5: Quantities and qualities of the pseudo labels at each iteration of PIEClass and Single-Ve View ST on Yelp. Each bar shows the portion of correct and wrong pseudo labels to the entire corpus D, and the line shows the quality (accuracy) of pseudo labels.
settings, where only the label name of each class is utilized as supervision, inspiring results (Meng et al.. 2020; Mekala and Shang, 2020; Wang et al., 2021; Zhang al., 2021). LOTClass (Meng et al., 2020) fine-tunes an MLM- PLM for category prediction and generalizes the model with self-training. ConWea (Mekala and Shang, 2020) leverages seed words and contextualized embeddings to disambiguate the keywords for each class. XClass (Wang et al., 2021) utilizes keywords to obtain static representations of classes and documents and generates pseudo labels by clustering. ClassKG (Zhang et al., 2021) learns the correlation between keywords by training a GNN over a keyword co-occurrence graph. Single-View ST on Yelp. The bars represent the per- centage of correct and wrong pseudo labels to the entire corpus D, and the lines are their quality mea- sured by accuracy (the number of correct pscudo labels over the total number of pseudo labels). We can observe that, Single-Views View ST progressively in- creases the number of pseudo labels but the quality of pseudo labels drops quickly, while PIEClass can keep the quality of pseudo labels during the expan- sion process. The number of pseudo labels does not increase much in the last two iterations, because PIEClass does not blindly expand the pseudo labels with potential errors. By utilizing two PLM fine- tuning methods and model ensemble, PIEClass only includes the most confident pseudo labels to ensure the quality, which contributes to its superior performance. More results are in Appx C.5.
studied and achieve
tently outperforms it. The reason is that the quality of pseudo labels becomes more crucial when the number of pseudo labels increases. Therefore, the performance of Single View Self-Training does not increase much during the iterative process due to its eтror accumulation problem, while the perfor- mance of PIEClass is increasing much faster. For efficiency, we the number of iterations to 5 or 3 except for 20News, but running more iterations may further improve the results.
Figure 5 shows the quantity and quality of the pseudo labels at each iteration by PIEClass and
However, these methods only depend on static keyword features, leading to noisy pseudo-labeled documents for classifier training. LOPS (Mekala et al., 2022) studies the order of pseudo label selection with learning-based confidence scores. concurrent work MEGClaas (Kargupta et al., 2023) studies how different text granularities can mutually enhance each other for document-level classification.
Data programming is another line of work on weak supervision. These methods either require do- main knowledge to provide heuristic rules (Chatter- jee et al., 2018) or some labeled samples to induce labeling functions (Varma and Ré, 2018; Pryzant et al., 2022), both (Maheshwari et al., 2021, 2022; Awasthi al., 2020). In this paper, fo- cus on training text classifiers with extremely weak supervision, i.e., using only the label names as su- pervision, so we do not compare with data program- ming methods that require additional knowledge
5 Related Work

---

like textual patterns and keyword lists (Ren al., 2020).
classification task where PIEClass achieves similar performance fully-supervised baseline.
There are three future directions that can be ex- plored. First, we can extend our method to other forms of text data (e.g., social media) and other ab- stract classes (e.g., stance detection, morality classi- fication) that require deeper text understanding and keyword-driven methods will likely fail. Second, PIEClass can be integrated with keyword-baa based methods as two types of training signals to further improve the performance of seakly--upervised text classification. Third, the idea of PIEClass is also generalizable to other text mining tasks with lim- ited supervision, such as named entity recognition and relation extraction.
Prompt-Based Learning PLMs (Devlin et al.. 2019; Radford et al., 2019; Liu et al., 2019) have shown superior performance on various down- stream tasks through fine-tuning with task-specific data. Some papers show that PLMs can learn generic knowledge during the pre-training stage and design cloze-style prompts to directly probe its knowledge without fine-tuning (Petroni et al., 2019; Davison et al., 2019; Zhang et al., 2020). Later, task-specific prompts are used to guide PLM fine-tuning and perform well in low-resource setting for several tasks, such as text classifica- tion (Han et al., 2021; Hu et al., 2022), relation ex- traction (Chen al., 2022), and entity typing (Ding et al., 2021; Huang et al., 2022). Recent works use prompts for keyword or rule discovery (Zeng et al., 2022; Zhang et al., 2022a) or directly prompt large PLMs for weak supervision (Smith et al., 2022). To mitigate the human efforts in prompt engineering, researchers also study automatic methods includ- ing prompt search (Shin et al., 2020; Gao al., 2021) and prompt generation (Guo et al., 2022; Deng et al., 2022). Soft prompts are also proposed by tuning some randomly initialized vectors to- gether with the input (Zhong et al., 2021; Li and Liang, 2021; Lester et al., 2021). Lang et al. (2022) also shows that the co-training method can ben- efit prompt-based learning in few-shot setting. Besides standard prompting methods for MLM- based PLMs, prompting methods for discrimina- tive PLMs are also studied on few-shot tasks (Xia et al., 2022; Yao et al., 2022; Li et al., 2022).
Limitations
In this paper, propose PIEClass, general method for weakly-supervised text classification. We introduce an iterative ensemble framework by combining two standard PLM fine-tuning meth- ods for noise robustness. Despite its effectiveness shown in the experiments, there is still room for im- provement. For example, our learning framework can be integrated with other PLM fine-tuning meth- ods and noise-robust training objectives (Zhang and Sabuneu, 2018; 88; Meng al., 2021). Besides, our method uses PLM prompting to acquire pseudo labeled documents. As we only use several popu- lar corpora, verbalizers, and prompts for this task, it may require additional efforts to find verbaliz- ers/promptt if working on other domains. Finally, our iterative pscudo label expansion framework re- quires access to number of unlabeled documents, so it may perform worse if the corpus is too small.
6 Conclusion and Future Work
Acknowledgments
In this paper, we study the task of weakly- supervised text classification that trains classi- fier using the label names of target classes as the only supervision. To overcome the limitations of existing keyword-driven methods, we propose PIEClass which consists of two modules: (1) an initial pseudo label acquisition module using zero- shot PLM prompting that assigns pseudo labels based on contextualized text understanding, and (2) a noise-robust iterative ensemble training module that utilizes two PLM fine-tuning methods with model ensemble expand pseudo labels while en- suring the quality. Extensive experiments show that PIEClass can achieve overall better performance than strong baselines, especially on the sentiment
Research was supported in part by US DARPA KAIROS Program No. FA8750-19-2-1004 and INCAS Program No. HR001 12100165, National Science Foundation IIS- 19-5155,,,,, 1IS-17-41317. and IIS 17-04532, and the Molecule Maker Lab Institute: An AI Research Institutes program sup- ported by NSF under Award No. 2019897, and the Institute for Geospatial Understanding through an Integrative Discovery Environment (I-GUIDE) by NSF under Award No. 21 2118829. Any opinions, findings, and conclusions or recommendations ex- pressed herein are those of the authors and do not necessarily represent the views, either expressed or implied, of DARPA or the U.S. Govermment.

---

References
Han Guo, Bowen Tan. Zhengzhong Liu, Eric Xing. and Zhiting Hu. 2022. Efficient (soft) q-learning for text generation with limited good data. In EMNLP Findings.
Eugene Agichtein and Luis Gravano. 2000. Snowball: extracting relations from large plain-text collections. In Digital library.
Han, Weilin Zhao, Ning Ding, Zhiyuan Liu, and Maosong Sun. 2021. Ptr: Prompt tuning with rules for text classification. arXiv preprint arXiv:2 11259.
Abhijeet Awasthi, Sabyasachi Ghosh, Rasna Goyal, and Sunita Sarawagi. 2020. Learning from rules general- izing labeled exemplars. In ICLR.
Sonia Badene, Kate Thompson. Jean-Pierre Lorré, and Nicholas Asher. 2019. Data programming for learn- ing discourse structure. In ACI.
Shengding Hu, Ning Ding, Huadong Wang, Zhiyuan Liu, Jingang Wang, Juanzi Li, Wei Wu, and Maosong Sun. 2022. Knowledgeable prompt-tuning: Incor- porating knowledge into prompt verbalizer for text classification. In ACL.
Avrim Blum and Tom M. Mitchell. 1998. Combining la- beled and unlabeled data with co-training. COLT.
Jiaxin Huang, Yu Meng, and Jiawei Han. 2022. Few- shot fine-grained entity typing with automatic label interpretation and instance generation. In KDD.
Ming-Wei Chang, Lev-Arie Ratinov, Dan Roth, and Vivek Srikumar. 2008. Importance semantic repre- sentation: Dataless classification. In AAAI.
Priyanka Kargupta, Tanay Komarlu, Susik Yoon, Xuan Wang, and Jiawei Han. 2023. MEGClass: Text classification with extremely weak supervision via mutually- text granularities. In EMNLP.
Oishik Chatterjee, Ganesh Ramakrishnan, and Sunita Sarawagi. 2018. Data programming using continuous and quality-guided labeling functions. In AAAI.
Samuli Laine and Timo Aila. 2017. Temporal ensem- bling for semi-supervised learning. ICLR.
Xiang Chen, Ningyu Zhang, Xin Xie, Shumin Deng. Yunzhi Yao, Chuanqi Tan, Fei Huang, Luo Si, and Huajun Chen. 2022. Knowprompt: Knowledge- aware prompt-tuning with synergistic optimization for relation extraction. In WWW.
Hunter Lang, Monica N. Agrawal, Yoon Kim, and
David A. Sontag, 2022. Co-training improves prompt-based learning for large language models. In ICML.
Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: Pre- training text encoders as discriminators rather than generators. In ICLR.
Ken Lang. 1995. Newsweeder: Learning to filter net- news. In ICML.
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power scale for parameter-efficient prompt tuning. In EMNLP.
Joe Davison, Joshua Feldman, and Alexander Rush. 2019. Commonsense knowledge mining from pre- trained models. In EMNN.P-JCNNIP.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. ACL-IJCNLP.
Mingkai Deng, Jianyu Wang. Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric P. Xing, and Zhiting Hu. 2022. Riprompt: Optimizing discrete text prompts with reinforcement learning. In EMNLP.
Zicheng Li, Shoushan Li, and Guodong Zhou. 2022. Pre-trained token-replaced detection model few- shot learner. In COLING.
Jacob Devlin, Ming-Wei. Wei Chang, Kenton Lee, and Kristina Toutanova. 2019, Bert: Pre-training of deep bidirectional transformers for language understand- ing. In NAACI-HILT.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselín Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. ArXiv, abs/1907 1etp/997..19999
Ning Ding, Yulin Chen, Xu Han, Guangwei Xu, Pengjun Xie, Hai-Tao Zheng, Zhiyuan Liu, Juanzi Li, and Hong-Get Gee Kim. 2021. Prompt-learning for fine-grained entity typing. arXiv preprint arXiv:2 108. 10604.
Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. ICLR.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng. and Christopher Potts. 201 I. Learning word vectors for sentiment analysis. In ACL.
Evgeniy Gabrilovich and Shaul Markovitch. 2007. Computing semantic relatedness using wikipedia- based explicit semantic analysis. In IJCAL.
Ayush Maheshwari, Oishik Chatterjee, Krishnateja Kil- lamsetty, Ganesh Ramakrishnan, and Rishabh Iyer. 2021. Semi-supervised data programming with sub- set selection, In ACL-IJCNLP Findings.
Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In ACL-IJCNLP.

---

Ayush Maheshwari, Krishnateja Killamsetty, Ganesh Ramakrishnan, Rishabh Iyer, Marina Danilevsky, and Lucian Popa. 2022. I Learning to robustly aggregate labeling functions for supiriueereiee data program- ming. In ACL Findings.
Wendi Ren, Yinghao Li, Hanting Su, David Kartchner, Cassie Mitchell, and Chao Zhang. 2020. Denoising multi-source weak supervision for neural text classi- fication. In EMNLP Findings.
Evan Sandhaus. 2008. The New York Times Annotated Corpus.
Julian McAuley and Jure Leskovec. 2013. Hidden fac- tors and hidden topies: understanding rating dimen- sions with review text. In RecSys" 13.
Timo Schick, Helmut Schmid, and Hinrich Schütze. 2020. Automatically identifying words that can serve as labels for few-shot text classification, In COLING.
Dheeraj Mekala, Chengyu Dong, and Jingbo Shang. 2022. LOPS: Learning order inspired pseudo-label selection for weakly supervised text classification. In EMNLP Findings.
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. In EMNLP.
Dheeraj Mekala and Jingbo Shang, 2020. Contextu- alized weak supervision for text classification. In
Kai Shu, Subhabrata Mukherjee, Guoqing Zheng. Ahmed Hassan Awadallah, Milad Shokouhi, and Su- san Dumais. 2020. Learning with weak supervision for email intent detection. In SIGIR.
ACI.
Yu Meng, Jiaming Shen, Chao Zhang, and Jiawei Han. 2018. Weakly-supervist neural text classification. In CIKM.
Ryan Smith, Jason Alan Fries, Braden Hancock, and Stephen H. Bach. 2022. Language models in the loop: Incorporating prompting into weak supervision. ArXiv, abs/2205.023. 023188.
Yu Meng, Yunyi Zhang, Jiaxin Huang. Xuan Wang. Yu Zhang, Heng Ji, and Jiawei Han. 2021. Distantly- supervised named entity recognition with noise- robust learning and language model augmented self- training. In EMNLP.
Yangqiu Song and Dan Roth. 2014, On dataless hierar- chical text classification. In AAAI.
Yu Meng, Yunyi Zhang, Jiaxin Huang, Chenyan Xiong, Heng Ji. Chao Zhang, and Jiawei Han. 2020. Text classification using label names only: A language model self-training approach. In EMNLP.
Duyu Tang, Bing Qin. and Ting Liu. 2015. Document modeling with gated recurrent neural network for sentiment classification. In EMNLP.
Fangbo Tao, Chao Zhang, Xiusi Chen, Meng Jiang. Tim Hanratty, Lance Kaplan, and Jiawei Han. 2018. Doc2cube: Allocating documents to text cube with- out labeled data. In ICDM.
OpenAI. 2023. Gpt-4 technical report. ArXiv, abs/2303.08774.
Seongmin Park and Jihwa Lee. 2022. LIME: Weakly- supervised text classification without seeds. In COL- ING.
Paroma Varma and Christopher Ré. 2018. Snuba: Au- tomating weak supervision to label training data. Proc. VLDB Endow., 12:223-236
Shichao Pei, Lu Yu, Guoxian Yu, and Xiangliang Zhang. 2022. Graph alignment with noisy supervision. In www.
Zihan Wang, Dheeraj Mekala, and Jingbo Shang. 2021. X-class: Text classification with extremely weak su- pervision. In NAACI.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowl- edge bases? In EMNLP-IJCNLP?
Mengzhou Xia, Mikel Artetxe, Jingfei Du, Danqi Chen, and Ves Stoyanov. 2022. Prompting electra: Few- shot learning with discriminative pre-trained models. In EMNLP.
Reid Pryzant, Ziyi Yang. Yichong Xu, Chenguang Zhu, and Michael Zeng. 2022. Automatic rule induction for efficient semi-supervised learning. In EMNLP Findings.
Zichao Yang. Diyi Yang. Chris Dyer, Xiaodong He. Alexander J. Smola, and Eduard H. Hovy. 2016. Hi- erarchical attention networks for document classifi- cation. In NATCL-HLT.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Yuan Yao, Bowen Dong, Ao Zhang, Zhengyan Zhang, Ruobing Xie, Zhiyuan Liu, Leyu Lin, Maosong Sun, and Jianyong Wang. 2022. Prompt tuning for dis- criminative pre-trained language models. In ACL Findings.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In EMNLP.
Ziqian Zeng, Weimin Ni, Tianqing Fang, Xiang Li, Xinran Zhao, and Yangqiu Song. 2022. Weakly pervised text classification using supervision signals from a language model. In NAACL Findings.
Alexander Ratner, Christopher De Sa, Sen Wu, Daniel Selsam, and Christopher Ré. 2016. Data program- ming: Creating large training sets, quickly. In NIPS.

---

Lu Zhang, Jiandong Ding, Yi Xu, Yingyao Liu, and Shuigeng Zhou. 2021. seaely-superviied text clas- sification based on keyword graph. In EMNLP.
Then, given the training samples {(d,, c:) the PLM model and the randomly initialized classi- fication head (normally a single linear layer) are optimized with the cross-entropy loss:
Rongzhi Zhang. Yue Yu, Pranav Shetty, Le Song, and Chao Zhang, 2022a. coompt-based rule discovery and boosting for interactive super-sup leamn ing. In ACL.
Chead Σ log p(c/d))
(6)
Because the PLM is not pre-trained for any spe- cific downstream task, the [CLS] token embedding does not contain the necessary information if not fine-tuned. Besides, the randomly initialized clas- sification head also needs to be trained. Therefore, normally the head token fine-tuning needs a sub- stantial amount of labeled data for training. Other- wise, the model can casily overfit the training data given large number of parameters to update. For example, existing weakly cation methods use class-indicative keywords assign pseudo labels to documents which are then used to fine-tune a PLM using its [CLS] token.
Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text clas- sification. In NIPS.
Yu Zhang, Shweta Garg, Yu Meng, Xiusi Chen, and Jiawei Han. 2022b. Motifclass: Weakly supervised text classification with higher-order metadata infor- mation. In WSDM.
Yunyi Zhang, Fang Guo, Jiaming Shen, and Jiawei Han. 2022c. Unsupervised key event detection from mas- sive text corpora. In KDD.
text classifi-
Yunyi Zhang, Jiaming Shen, Jingbo Shang, and Jiawei Han. 2020. Empower entity expansion via lan- guage model probing. In ACL.
Prompt-Based Fine- TanneT.nngg. To close the gap be- tween PLM's pre- pre-training task and the downstream applications, prompt-based fine-tuning is proposed to convert the input and output of the downstream task to a similar form of the pre-training task. Be- cause of the similarity between pre-training and fine-tuning tasks, prompt-based fine-tuning only needs a small set of samples to achieve competi- tive performance with head token fine- tieettunigg. For common PLMs pre-trained with masked language modeling (c.g.. BERT, RoBERTa), prompt- based fine-tuning uses template to convert an input se- quence into a cloze-style task. Each class also associates with one or more verbalizers, and PLM will predict the likelihood of each verbalizer for the masked position. For example, for sentiment classification task, a template 7-M can transform document d as:
Zhilu Zhang and Mert R. Sabuncu. 2018. Generalized cross entropy loss for training deep neural networks with noisy labels. In NIPS.
Zexuan Zhong, Dan Friedman, and Danqi Chen. 2021. Factual probing is [MASK]: Learning vs. learning to recall. In NAACL.
Dawei Zhu, Xiaoyu Shen, Marius Mosbach, Andreas Stephan, and Dietrich Klakow. 2023. Weaker than you think: A critical look at weakly supervised learn- ing. In ACL.
A Preliminaries on PLM Fine-Tuning
Recently, Transformer-based large language mod- els achieve remarkable performance on down- stream tasks by first pre-training on large corpora to capture generic knowledge and then finettning with task-specific data. There generally two fine-tuning strategies for the sequence classification tasks: head token fine-tuning and prompt-based fine-tuning. See Figure for some examples.
T TMIM (d) It was [MASK].
Given MMM (d) as input, the pre-trained PLM and its pre- tre-trained MLM head f will generate a proba- bility distribution over its vocabulary, indicating the likelihood of cach token appcaring in the masked position.
Head Token Fine-Tuning, PLMs like BERT and RoBERTa add additional [CLS] token at the beginning of the input sequence and it can be fine- tuned for sequence classification tasks to capture the information of the entire sequence. To fine-tune for downstream task, the contextualized represen- tation of the [CLS] token CLS document is fed into a linear classification head g to get
p(wld) = Sofimax(f(h. (h(Mk)))
(7)
The probability of predicting class c, assuming its label name ((c) as its only verbalizer, is the probability of its verbalizer pd)(c)ld). During fine- tuning, the PLM and its MLM head can be trained with standard cross-entropy loss.
p(cld) = (o
(5)

---

B Experiment Setup
for initial pseudo label acquisition is l 10%. During the iterative process, the coefficient for the increasing size of pseudo labels is & = 20%, except for NYT-Topics and NYT-Fine which are highly imbalanced, for which we set 8 = 35% to ensure enough pseudo samples for the rare classes. Notice that this parameter can be decided by just observing the model's intermediate outputs instead of using any labeled data. The threshold of confidence score is p 0.95. We randomly sample r subsets of size g = % of the candidate pseudo labels for prompt-base based fine-tuning and model ensemble. We set the number of full iterations T 1/s, which is for AGNews, Yelp. IMDB, and Amazon and 3 for NYT-Topies and NYT-Fine; for 20News that is harder, we run until the number of pseudo labels does not increase, which takes full iterations.
B.1 Datasets
Table 4 shows the label names and prompts used for each dataset.
B.2 Compared Methods
WeSTClass (Meng et al., 2018) trains CNN classifier with pseudo documents generated based on keyword embeddings and then applies self-training on the unlabeled documents.
Con Wea (Mekala and Shang, 2020) utilizes pre-trained language model get pseudo labels using contextualized representations of keywords. It then trains a text classifier and uses the results to further expand the keyword sets.
LOTClass (Meng et al., 2020) uses a pre-trained language model to discover class-indicative key- words and then fine-tunes the PLM using self- training with the soft labeling strategy.
C Additional Experiments
C.1 Comparison with Few-Shot Classifiers
XClass (Wang et al., 2021) first expands the class-indicative keyword sets help estimate class and document representations. Then, clus- tering algorithm used to generate pseudo labels for fine-tuning a text classifier.
We follow similar settings in Meng et al. (2020) and Zhu et al. (2023) to see how many human labeled samples are needed to achieve similar performance PIEClass. Here, ELECTRA- base fine-tuned with few labeled samples in a fully supervised way and compared with PIEClass [ELECTRA+ELECTEE
ClassKG (Zhang et al., 2021) constructs key- word graph with co-occurrence relations and self- train sub-graph annotator, from which pseudo labels are generated for classifier training and the predictions are used to update keywords itera- tively.
Since PIEClass does not use any annotated validation data under the weakly- supervised setting, for a fair comparison, we do not provide validation sets for the few-shot classifiers as well. Instead, we directly report the results of the last checkpoint in the training process and we do not see obvious model overfitting the train- ing set. The results in Macro-Fl on AGNews and IMDB are shown in Table 5. We can see that it needs around 150 labels per class on AGNews (to- tally 600) and 500 labels per class on IMDB (totally 1000) to achieve similar results to PIEClass, which requires non-trivial amount of labeling efforts. In contrast, PIEClass only needs one label name for each class supervision and thus drastically reduces the needs of human efforts.
B.3 Implementation Details
We use pre-trained ELECTRA BEECTA---aa---iiii discrimirrmm,,,rr, BERT-baase-ncaseed and RoBERTa- base as the backbone models for the corresponding versions of PIEClass. The classification head for head token fine-tuning is single linear layer. The training batch size is 32 for both head token fine-tuning and prompt-based fine-tuning. We train 5 epochs and use A (Loshchilov and Hutter, 2017) as the optimizer for all the fine-tuning tasks. The peak learning rate is 1e - 5 for prompt-based fine- tuning of RoBERTa and 2c- for prompt-based fine-tuning of ELECTRA and all head token fine- tuning, with linear decay. For Yelp and IMDB that have only two classes, to avoid overfitting when the number of pseudo labels is small, we freeze the first 11 layers of the PLM for fine-tuning in the first several iterations and only fine-tune the full model for the final classifier. The model is run on one NVIDIA RTX A6000 GPU. The threshold
C.2 Impact of Backbone PLMs
We include the ELECTRA+BERT version to com- pare with baselines that also use BERT the back- bone model for classifier training, and also the RoBERTa+ROBERTA version which only uses one single MLMM-based model without access to ELEC- TRA. Also, ELECTRA is pre-trained with the same data as BERT, so using it does not give the model

---

Table 4: Label names and prompts used for each dataset.
Dataset
Label Names
Prompt
AGNews
politics, sports, business, technology
[MASK] News: <doc>
20News
computer, sports, science, politics, religion
[MASK] News: <doc>
business, politics, sports, health, education, estate, arts, science, technology
NYT-Topies
[MASK] News: <doc>
music, bascball, business, abortion, military, football, television, economy, dance, soccer, cosmos, surveillance, golf, law, basketball, budget, movics, stocks, gun, energy, environment, hockey, healthcare, immigration, tennis, gay
NYT-Fine
[MASK] News: <doc>
Yelp
good, had
<doc> It was [MASK].
IMDB
gond, had
<doc> It was [MASK].
Amazon
good. bad
<doc> It was [MASK].
Table 5: Macro-Fl of vanilla few-shot classifiers with different numbers of labels per class compared with PIEClass (ELECTRA+ELECTRA),
labels per class
100
150
250
500
PIEClass
AGNews IMDB
0.875
0.885
0.884
0.900
0.915
0.925
0.933
0.931
Table 6: Performance of original XClass, XClass with ELECTRA as the final classifier, and two versions
PIEClass measured by Micro/Macro-Fl.
Methods
AGNews
20News
NYT-Topies
Yelp
XClass
0.857/0.857 0.838/0.837
0.786/0.778
0.790/0.686
0.900/0.900
XClas-ELECTRA
0.792/0.784
0.787/0.685
0.903/0.903
PIEClass
ELECTRA+BERT
0.884/0.884
0.789/0.791
0.807/0.7 710
0.919/0.919
ELECTRA+ELECTRA
0.884/0.884
0.816/0.817
0.832/0.763
0.957/0.957
explicit advantages. Both versions achieve strong enough performance on all datasets, especially on sentiment classification, which demonstrates the effectiveness of PIEClass. We did not include BERT+BERT version because RoBERTa, as pow- erful variant of BERT, is used more popularly for prompting with MLM.
experiments. Table shows the Micro/Macro-FI scores, with ELECTRA bringing small improve- ments to XClass' s performance on two datasets and PIEClass (ELECTRA+ELECTRA) consistently outperforms XClass and XClas-EELECTRA.
We also run XClass by using ELECTRA as the backbone of its final classifier to study the effects. Also note that we only change the backbone of the final classifier of XClass and still use BERT for its pseudo label assignment step, because we empiri- cally find that switching to ELECTRA for its entire framework drastically decreases its performance: we get 0.77 on Yelp and only 0.43 on AGNews. This also shows that XClass is not generalizable to different types of PLMs while PIEClass is ap- plicable to various types PLMs as shown in our
C.3 Parameter Sensitivity
We study the parameter sensitivity of PIEClass by varying the following parameters on IMDB: the threshold g for the initial pseudo labels, the mini- mum probability threshold p during the iterations. and the number of prompt-based classifiers Fig- ure shows performance measured by Micro-F1. We find that overall PIEClass not sensitive to these parameters.

---

1.00
1.00
0.95
0.93
0.931
0.928
0.929
0.924
0.93
0.931
0.95
0.925
0.931
0.929
0.952
0.00
0.50
0.00
0.85
0.85
0.05
0.1
0:15
0.2
0.85
0.9
0.95
4 of prompt based classifiers (default 3)
initial threshold 1 (default 0.1)
probability p (default 0.95)
Figure 6: Performance of PIEClass IMDB when varying different parameters.
Table 7: Macro-F1 on AGNews with constant thresh- old f2 =00% for pseudo label selection.
Table 8: Performance of PIEClass and zero zero-shot prompt- ing of ELECTRA with different sets of verbalizers, mea- sured by Micro-Fl/Macro-Fl.
Iteration
3
Verbalizers
Methods
Yelp
Macro-F1 0.847
0.815
0.775
0.780
0.780
ELECTRA (0-shot)
0.820/0.820
0.803/0.802
good/bad
PIEClass
0,957/0,957
0.931/0.931
great/ terrille
ELECTRA (0-shot)
0.880/08880
0.844/0.844
C.4 Effects of Using Constant Threshold f;
PIEClass
0.959/0,959
0.933/0.933
We increase the threshold 4 in each iteration to gradually increase the number of selected confident pseudo labels during the iterative process. Because using constant threshold samples almost the same in the iterative process and the classifiers overfit more the limited number of pseudo samples. We tried on AGNews by keeping the threshold constantly equal to 20%. The perfor- mance of the classifier in each iteration shown in Table 7, where we can see the performance first drops and then stays almost constant. In this work, we simply use linearly increasing threshold. In fact, more advanced curriculum learning strategies can be applied better fit the distribution of predic- tion scores, e.g., self- self-pacing function (Pei al., 2022).
tokens. For example, the label name "religion" of 20News is tokenized by RoBERTa into two tokens, "rel" and "igion". Therefore, prompting RoBERTa for multi-token label names requires substantially more work by inserting multiple [MASK] tokens into the template and iteratively predicting the masked tokens. On the other hand, prompting ELECTRA can easily handle multi-token label names (Xia et al., 2022), because the label names are directly encoded in the input. Assume that label name I(c) is tokenized into several pieces [(c) {w1,--.. ([((() }. We can estimate the prob- ability of its being original by taking the average of the probabilities of each token,
make the pscudo
[e]l
C.5 Additional Results on the Study of Iterative Process
p(((c)ld)
Σ p(w,ld)
(8)
[(()]]
Figure and Figure show more results for study- ing the iterative process of PIEClass (c.f. Sec 4.4). We can observe similarly that PIEClass can ensure higher quality pseudo labels during the iterative process compared with Single-Vier ST.
Templates for PLM Prompting One limitation of PLM prompting is that its performance is re- lated the quality of the templates and verbal- izers. In this work, we directly use the prompts for sentiment classification and news topic classi- fication from previous studies (Gao et al., 2021; Xia et al., 2022) without any further tuning. To mitigate the human efforts on prompt engineering, some automatic methods proposed to optimize the prompts, including prompt search (Shin et al., 2020; Gao et al., 2021) and prompt generation (Guo et al., 2022: Deng et al., 2022).
Discussions on PLM Prompting
Handling Multi-Token Label Names As shown in the experiment results (Table 2), the perfor- mance of prompting with MLM-based PLMs such as RoBERTa is affected by the tokenizer, because the MLM classification head cannot naturally han- dle verbalizers (i.e., label names) with multiple
The selection of verbalizers can also affect the performance of prompt- based methods for PLMs.

---

AGNews
IMDB
Amazon
0.05
1.00
1,00
PIEClass
PlEClass
SingleVView
Singl=View ST
Single-Viee ST
Mi0o-11
Micro-F)
Micre-F1
0.65
0.50
0.30
0.75
0.50
0.90
Number
full
Figure 7: Performance of PIEClass and Single- View ST by varying the number of full iterations.
AGNews (PIEClass) 200
20Neww (PIEClass
IMDB (PIEClass)
Amazon (PIEClass)
100-
96.
024
200-
14.
100-
70-
Desnagge
Pavaragg
70
60-
acaurtey
Iterations
Iteations
(Single View ST)
20News (Single
(Single
(Single View ST)
100
91
100-
305
100
91h
100
Parsetage
Pararage
5
Teeco ons
Iterators
herations
Figure 8: Quantities and qualities of the pseudo labels at each iteration of PIEClass (top) and the Single- Single-View View ST ablation (bottom).
In this paper, we directly use the label names from previous works weakly-supervised text classi- fication main results. Here, also try different set of verbalizers for sentiment classifica- tion, "great" and "terrible", that are used in previ- ous papers studying prompt-based methods (Gao et al., 2021; Xia et al., 2022). Table 8 shows the performance of ELECTRA (0-shot) and PIEClass with the two sets of verbalizers. We can see that, by changing the verbalizers, the zero-shot prompt- ing performance increases by large amount and even achieves comparable results to the keyword- driven baselines on Yelp. PIEClass also performs better with the new verbalizers. Therefore, opti- mizing verbalizers could promising next step for prompt-based text classification by verbalizer search (Gao et al., 2021; Schick et al., 2020) or learning verbalizer-class correlation (Huang et al., 2022).


Reviews:
[{"review_id": "hBou6d5WM9", "reviewer": "Reviewer_oKTC", "summary": "", "questions": "1. It seems there is a significant overlap with the data programming literature. However, I cannot find its mention in the paper except Line 555.\n\n2. A complete 'Algorithm' describing the whole process would be helpful to understand. Line 324-336 describes but it can be more useful in the form of an algorithm.\n\n3. For the uninitiated, head-token fine-tuning should be explained properly in the main paper itself. Figure 1 caption attempts to explain but IMO it should be described more formally.\n\n4. Line 325 : \"text classifier, $F_0^i : D → C$,using the current pseudo labels $P^{i−1}$ in a fully- supervised way\" . Should it be $P^i$ ?\nIn the same paragraph, what does $0$ in  $F_0^i$ refers to ? \n\n5. In Line 340, for the sake of clarity, it should be mentioned that  $q \\in [0,1]$. In line 339, is random sampling done with a replacement ?\n\n6. Line 339: \"Because the noisy labels are unlikely to be sampled repeatedly into different subsets\". Is it because you are selecting only top-k confident predictions from each classifier ?\n\n7. The paper should contain atleast some technical details about ELECTRA in the main paper itself since its a key component for reducing noise by PLMs. \n\n8. What happens to the less-confident predictions in further iterations. Are those instances dropped from the set ?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive comments on our paper! We address your questions as follows and will clarify them in the revision:\n\n**1. Missing discussion to data programming works:**  \nThanks for pointing out this line of work. We did not discuss data programming methods in detail because they either **require domain knowledge** [1] to provide heuristic rules or **some labeled samples** [2, 3] to induce labeling functions (or both [4, 5, 6]). Because we focus on training text classifiers with extremely weak supervision, i.e., **using only the label names as supervision**, we did not compare our method with the data programming methods that require additional knowledge as supervision (e.g., keyword lists for AGNews, textual patterns for IMDB, and keywords from multiple views/aspects for Yelp [7]). Although Snuba [2] also introduced an iterative learning process, it is under the semi-supervised setting where a set of labels is available and its iterative feedback is used for generating more heuristics.\n\nWe agree with you that this line of work is related to the weak supervision setting discussed in our paper, and **we will add related discussion in the revision**. In fact, our method can also be considered as using PLM prompting as the labeling function which requires minimal amount of knowledge for prompt designing, compared with the heuristic rules and/or labeled samples for data programming that require some level of domain expertise. We propose an iterative ensemble approach that can effectively mitigate the impact of noisy labels for classifier training in the extremely weakly supervised setting without any labeled data.\n\n**2. Add a complete ‘Algorithm’:**  \nDue to the space limitation, we did not put an ‘Algorithm’ or pseudocode in the methodology section. We agree with you that having such can help readers better understand the methodology and we will add it in the revision.\n\n**3. Discussion on uninitialized head-token fine-tuning:**  \nWe have more detailed discussion on the preliminaries of head-token fine-tuning and prompt-based fine-tuning in Appendix A due to the space limitation of the main content. We will move those into the main content when possible in the revision.\n\n**4. Notations in Line 325-326:**  \n$P^{i-1}$ is defined as the pseudo label pool constructed in iteration $i-1$, so it is also the pseudo label available at the starting of iteration $i$ and is thus used as the training label for the classifier $F^i_0$. The 0 in the subscription of $F^i_0$ is just to represent the head-token classifier in each iteration and we can also name it as $F^i_{head}$. However, since later we will ensemble its results with $r$ prompt-based classifiers which we name as $F^i_1$ to $F^i_r$, we therefore give it the number zero so that (1) it shows the head-token fine-tuning happens before prompt-based fine-tuning and (2) the consistency of numbering makes it cleaner in the ensemble step Eq 4.\n\n**5. Parameter q and random sampling:**  \nEach random sampling is done without replacement. Thanks for pointing this out and we will clarify it in the revision. \n\n**6. Line 339, noise during random sampling:**  \nIt is because the sampling ratio (i.e., $q$) is kept small and the ratio of noisy labels is small compared with the correct labels as we only take top predictions to ensure quality. \n\n**7. Technical details of ELECTRA:**  \nWe agree that some technical details of ELECTRA, or in general discriminative PLMs, will better explain our methodology. Due to the limited space, we currently have an overall introduction to ELECTRA (Line 221-231) and citations to the original paper and related papers that use prompting with ELECTRA (see Clark et al., 2020, Xia et al., 2022, Yao et al., 2022, Li et al., 2022, in the submitted paper). We will add more details in the main content in the revision if there is space.\n\n**8. Less-confident predictions:**  \nThe less confident predictions of the current iteration will only be left out from the pseudo label pool of current iteration, but will be re-examined in later iterations and added to later pseudo labels if it is qualified. Because throughout the iterative process, as more quality pseudo labels are available for classifier training, we expect the classifier can make better predictions during this process and wrongly predicted samples in earlier iterations have a chance to be corrected with more powerful classifiers in later iterations.\n\nWe would like to thank you again for your comments on our paper and we hope our clarifications help you better understand our method. We will also clarify the points you raised in the revision. We are happy to answer any remaining questions you may have.\n\n[1] Chatterjee et al., Data Programming Using Continuous and Quality-Guided Labeling Functions, in AAAI 2020.  \n[2] Varma and Re, Snuba: Automating Weak Supervision to Label Training Data, in VLDB 2018.  \n[3] Pryzant et al., Automatic Rule Induction for Efficient Semi-Supervised Learning, in EMNLP 2022.  \n[4] Maheshwari et al., Semi-Supervised Data Programming with Subset Selection, in Findings of ACL 2021.  \n[5] Maheshwari et al., Learning to Robustly Aggregate Labeling Functions for Semi-supervised Data Programming, in Findings of ACL 2022.  \n[6] Awasthi et al., Learning from Rules Generalizing Labeled Exemplars, in ICLR 2020.  \n[7] Ren et al., Denoising Multi-Source Weak Supervision for Neural Text Classification, in Findings of EMNLP 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper presents an interesting approach to avoid noise from PLMs using an ensemble approach.", "reasons_to_reject": "1. The methodology section is difficult to understand. (See Qs for the Authors).\n\n2. Discussion of related work seems to miss some important weak-supervision literature.", "questions_to_authors": "1. It seems there is a significant overlap with the data programming literature. However, I cannot find its mention in the paper except Line 555.\n\n2. A complete 'Algorithm' describing the whole process would be helpful to understand. Line 324-336 describes but it can be more useful in the form of an algorithm.\n\n3. For the uninitiated, head-token fine-tuning should be explained properly in the main paper itself. Figure 1 caption attempts to explain but IMO it should be described more formally.\n\n4. Line 325 : \"text classifier, $F_0^i : D → C$,using the current pseudo labels $P^{i−1}$ in a fully- supervised way\" . Should it be $P^i$ ?\nIn the same paragraph, what does $0$ in  $F_0^i$ refers to ? \n\n5. In Line 340, for the sake of clarity, it should be mentioned that  $q \\in [0,1]$. In line 339, is random sampling done with a replacement ?\n\n6. Line 339: \"Because the noisy labels are unlikely to be sampled repeatedly into different subsets\". Is it because you are selecting only top-k confident predictions from each classifier ?\n\n7. The paper should contain atleast some technical details about ELECTRA in the main paper itself since its a key component for reducing noise by PLMs. \n\n8. What happens to the less-confident predictions in further iterations. Are those instances dropped from the set ?", "paper_topic_and_main_contributions": "The paper proposes a weak-supervision technique wherein keywords (or deep features)  are leveraged for classification. The paper proposes to rely on output labels instead of the input sequence to guide the classification pipeline. It acquires labels via pre-trained language (PLM) model prompting. It performs an iterative process to train using pseudo labels by PLM and selects most confident predictions and trains a text classifier on those instances. To avoid propagation of incorrect labeling by PLMs, it use an ensemble of two PLM based classifier  as the final prediction.\n\n\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "The data programming also corresponds to the weak-supervision literature. None of the relevant papers from the data-programming domain seems to be discussed and/or compared in the paper. Refer few papers below:\n\na) Robust Data Programming with Precision-guided Labeling Functions. AAAI 2020: 3397-3404 \n\nb) Learning to Robustly Aggregate Labeling Functions for Semi-supervised Data Programming. ACL (Findings) 2022: 1188-1202 \n\nc) Semi-Supervised Data Programming with Subset Selection. ACL/IJCNLP (Findings) 2021: 4640-4651 \n\nd) Learning from Rules Generalizing Labeled Exemplars. ICLR 2020 \n\nThe similar iterative process is adopted by Snuba for selecting the most relevant rules for the text classification.\ne) Snuba: Automating Weak Supervision to Label Training Data, VLDB 2018 \n\nf) Automatic Rule Induction for Efficient Semi-Supervised Learning, EMNLP 2022"}}, {"review_id": "e0foDeHkNr", "reviewer": "Reviewer_dUCf", "summary": "", "questions": "1. Question A: this question is related to the first two reasons to reject. Have the authors compared performance with vanilla fine-tuning on a few labeled samples? Given that iterative training increases the training time (Figure 3), what if one chooses vanilla fine-tuning for fast training and devotes the saved time to manually annotating a small set of samples? What would be the performance gap to zero-shot learning methods? \n2. Question B:  at each iteration, is the F^{i}_{0} (L325) always freshly initialized? Or do the authors continue training the model from the previous iteration?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments on our paper! We are happy to know that you find our paper easy to follow and appreciate your recognition of the significance of our paper for weakly supervised and zero-shot text classification.\n\nWe would like to address your concerns on our paper as follows\n\n**1. Hyperparameter setting**  \nThanks for your constructive comments. We agree with you that using the same set of hyperparameters across all datasets makes our evaluation better align with practical scenarios. However, because the compared baselines also use different hyperparameters for different datasets, e.g., the number of epochs in LOTClass ([github](https://github.com/yumeng5/LOTClass)), the number of keywords per class and batch sizes in ClassKG ([github](https://github.com/zhanglu-cst/ClassKG)), we believe it is unavoidable for a fair comparison. The only parameter that is different in our experiments is the coefficient $s$ for the increasing pseudo label size, which is 35% for two NYT and 20% for others. This number is decided because of the unbalancing nature of the NYT corpus, and having enough pseudo samples for the rare classes can help the training process. In fact, unlike hyperparameters such as batch size or learning rate that do require labeled samples for tuning, **the hyperparameter $s$ can be tuned just by observing the model’s intermediate outputs instead of using labeled data**. For example, for the NYT-Fine dataset which is unbalanced across classes, if we set $s=20$%, the rare classes will have less than 3 pseudo labeled samples (only 1 for the most infrequent class). Therefore, we increase $s$ to 35% which gives at least 10 pseudo samples per class.\n\nFor the parameter sensitivity analysis in Appendix C.2, we observe similar trends on other datasets and thus select one dataset IMDB to illustrate it. We will clarify this in the revision. \n\nWe also follow your suggestion (a) and run PIEClass (ELECTRA+ELECTRA) on two NYT datasets with the same parameters as other datasets (i.e., $s=20$%). We compare its results with other extremely weakly-supervised methods in the following table (thus excluding WeSTClass and ConWea). Although the performance is lower than the original setting, PIEClass still achieves strong performance compared with other extremely weakly-supervised baselines.\n| **Methods** | **NYT-Topics** | **NYT-Fine** |\n|:-------------|:----------------|:---------------|\n|**LOTClass**|   0.671/0.436  |  0.150/0.202  |\n| **XClass**   |  0.790/0.686   |  0.857/0.674  |\n| **ClassKG**|   0.721/0.658  |  0.889/0.705  |\n| **PIEClass (s=20%)**|  0.758/0.719   |  0.898/0.689  |\n\n**2. Comparison with a few-shot learner**  \nWe agree with you that comparing a few-shot baseline with vanilla PLM fine-tuning is worth investigating. We thus follow similar settings in [1] and [2] to see how many human labeled samples are needed to achieve similar performance as PIEClass. Here, a PLM is fine-tuned with a few labeled samples in a fully-supervised way, and we use ELECTRA-base as the backbone model for a fair comparison with PIEClass (ELECTRA+ELECTRA). Since PIEClass does not use any annotated validation data under the weakly supervised setting, for a fair comparison, we do not provide validation sets for the few-shot classifiers as well. Instead, we directly report the results of the last checkpoint in the training process and we do not see obvious model overfitting on the training set. We perform 3 individual runs and take the average score to account for the effect of randomness in label sampling. The results in Macro-F1 ($\\pm$ standard deviation) on AGNews and IMDB are shown in the following table. We can see that it needs around 150 labels per class on AGNews (totally 600) and 500 labels per class on IMDB (totally 1000) to achieve similar results of PIEClass, which requires a non-trivial amount of labeling efforts. In contrast, PIEClass only needs one label name for each class as supervision and thus drastically reduces the needs of human efforts.\n| **# labels per class** | **100** | **150** | **250** | **500** | **PIEClass**|\n|:--|:--|:--|:--|:--|:--|\n|**AGNews**|87.5 ($\\pm$ 0.45)|88.5 ($\\pm$ 0.26)|-|-|88.4 ($\\pm$ 0.21)|\n|**IMDB**|90.0 ($\\pm$ 1.23)|91.5 ($\\pm$ 0.91)|92.5 ($\\pm$ 0.43)|93.3 ($\\pm$ 0.25)|93.1 ($\\pm$ 0.09)|\n\n**3. Standard deviation**  \nWhile many previous studies also do not report standard deviation, we agree that it is an important indicator on measuring the stability. Thanks for your suggestions on the formatting and we will take that into consideration in the revision. Empirically, we find that running PIEClass with different seeds only has very small influence on the final performance. With three individual runs of PIEClass (ELECTRA+ELECTRA) on IMDB, we get an averaged Macro-F1 score of 0.931 with standard deviation 0.00094.\n\n**4. Model initialization in each iteration**  \nThe classifiers are freshly initialized with the pre-trained weights and not continuously trained from the last iteration. We will clarify it in the revision.\n\nWe would like to thank you again for your constructive comments! Hope we have addressed your concerns and we are happy to answer any further questions you may have.\n\n[1] Zhu et al., Weaker Than You Think: A Critical Look at Weakly Supervised Learning, ACL 2023.  \n[2] Meng et al., Text classification using label names only: A language model self-training approach, EMNLP 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed method works well on various datasets, achieving strong performance without training on annotated data. This should establish a strong baseline for future work in zero-shot text classification.\n2. The paper is well-written and easy to follow, and the techniques used are sound.\n3. Overall, the experiments and results are solid.", "reasons_to_reject": "1. The practicability of this method is still somewhat questioned. There is a set of hyperparameters that need to be determined to apply the proposed method. It is unclear how to determine these values in practice. While appendix C2 discusses the parameter sensitivity, it is only done on one dataset (IMDB). It needs to be clarified about the sensitivity of other datasets. In addition, appendix B3 (L971 - L981) mentions that different values are needed for different datasets. These values can only be determined by accessing certain information about the datasets (e.g., class imbalance). On the other hand, the effort of getting this information (e.g., by manually checking some samples from the dataset) may instead be devoted to getting some labeled samples for training.\nThe concern of practicability can be addressed in either of the two ways. a) use a fixed set of hyperparameters across all datasets and report the results again. This would show that the model is not sensitive to the hyperparameters across all settings. b) use a small set of labeled data for validation purposes so that the optimal hyperparameters can be decided [1][2][3]. But then one should also compare with a simple baseline that trains on this set of labeled data [3].\n2. An additional aspect regarding practicability: the paper mentions that “Earlier studies train text classifiers in a fully -supervised manner that requires a substantial amount of training data” (L38-39). Is that true for PLMs like RoBERTa? The paper only contrasts their performance with models that trained on all samples (the fully supervised upper bound) but not with models that trained on a small set of samples (i.e., still supervised learning, but in a few-shot learning setting).\n3. In Table 2 & 3: the standard deviation should also be reported, given that the performance gaps among top-performing systems are small. There would be more space if the authors reported performance from a range of 1-100 (88.4 vs. 0.884). If there is still a lack of space, the authors can consider reporting only Marco-F1 in the main paper and moving Micro-F1 to the appendix. Also, the number of runs for each experiment should be reported.\n\n[1] Oliver et al., Realistic Evaluation of Deep Semi-Supervised Learning Algorithms\n[2] Perez et al., True Few-Shot Learning with Language Models\n[3] Zhu et al., Weaker Than You Think: A Critical Look at Weakly Supervised Learning", "questions_to_authors": "1. Question A: this question is related to the first two reasons to reject. Have the authors compared performance with vanilla fine-tuning on a few labeled samples? Given that iterative training increases the training time (Figure 3), what if one chooses vanilla fine-tuning for fast training and devotes the saved time to manually annotating a small set of samples? What would be the performance gap to zero-shot learning methods? \n2. Question B:  at each iteration, is the F^{i}_{0} (L325) always freshly initialized? Or do the authors continue training the model from the previous iteration?", "paper_topic_and_main_contributions": "This work proposes PIEClass, a zero-shot text classification approach that performs well on seven benchmark datasets. This paper aims to train a text classifier that categorizes a given text into one of the predefined classes without using any text-label pairs for training (hence zero-shot). PIEClass iteratively trains the classifier. In each iteration, two models jointly determine a set of examples, together with their pseudo-labels (similar to self-training). These selected examples will be used for training the models in the next iteration. The key contribution lies in combining powerful methods to push the limit of zero-shot text classification.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "please see the references mentioned reasons to reject."}, "extra_text": {}}, {"review_id": "4olbbYe4i1", "reviewer": "Reviewer_cc9r", "summary": "", "questions": "A. In the prompting-based fine-tuning phase, the dataset is split into multiple subsets to train multiple classifiers respectively. With little training sample, is it possible to cause overfitting of a single classifier and reduce its robustness?\nB. Why is the threshold t_i set to increase linearly?\nC. Why does the ELECTRA+ELECTRA method perform so poorly in the YELP dataset in Table 5?\nD. It seems that the benefits of the proposed pseudo label generation method are not analyzed in the experiment?\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments on our paper and your recognition of the novelty and significance of our work! We are also glad to know you find our paper well-organized and easy to understand. We address the questions you raised as follow:\n\n**1. Is it possible to have overfitting of a single classifier**  \nThanks for raising this question. Yes, it is possible that given the small number of pseudo samples, a single classifier may overfit, and empirically we find that even without subsampling but using all pseudo labels for training, the model still can overfit in the earlier iterations given the small number of labels. This actually motivates us to **apply model ensembling to improve robustness against overfitting**. By randomly sampling subsets of pseudo labels and training multiple individual classifiers, it is unlikely that all classifiers overfit to the same set of samples/features. Then, by taking the intersection of the top predictions of all classifiers, we can largely mitigate the impact of potential noise caused by overfitted models. This can also be shown in our experiment analysis that our method can keep the pseudo label quality during the iterative process and outperform the Co-training ablation which also utilizes two PLM fine-tuning methods but does not apply model ensemble.\n\n**2. Why the threshold $t_i$ increases linearly**  \nWe increase the threshold in each iteration to gradually increase the number of selected confident pseudo labels during the iterative process. This is because if we keep the threshold constant, the pseudo samples in the last iteration will be predicted confidently again, making the pseudo samples almost the same in the iterative process and the classifiers overfit more to the limited number of pseudo samples. We tried on AGNews by keeping the threshold constantly equal to 20%. The performance of the classifier in each iteration is shown in the following table, where we can see the performance first drops and then stays almost constant.\n| **Iteration #** | **1**| **2** | **3** | **4** | **5** |\n| :-- |:--|:--|:--|:--|:--|\n|**Macro-F1 w/ constant $\\mathbf{t}$**| 0.847 | 0.815 | 0.775 | 0.780 |0.780|\n\nIn this work, we simply use a linearly increasing threshold. In fact, more advanced curriculum learning strategies can be applied to better fit the distribution of prediction scores, e.g., a self-pacing function [1]. Here, we try to keep it simple as it is not the major claim of our work.\n\n**3. Low performance score in Table 5**  \nIt is a typo and thanks for catching it. The actual performance of ELECTRA+ELECTRA on Yelp should be 0.957, which can also be seen in the main results Table 2. We will correct it in the revision.\n\n**4. Benefits of pseudo label generation**  \nOverall, we use PLM prompting for pseudo label generation which is contextualized given the prompts and can go beyond static keyword-based features. The benefits of the proposed pseudo label generation can be seen in the following two parts in the experiment results. **First**, in Table 2, we show the performance of zero-shot PLM prompting as two baselines and can observe that prompting ELECTRA can already achieve strong performance on some datasets. For example, on NYT-Fine, it achieves 0.765 micro-F1 which is better than the simple weakly-supervised baselines. **Second**, we report the performance of the Two-Stage ablation in Table 3, which is directly fine-tuning the final classifier on the initial pseudo labels. We can see it achieves comparable results to the strong baselines on all datasets, which shows the effectiveness of the proposed pseudo label generation step.\n\nWe would like to thank you again for your comments! We appreciate your recognition of the importance of our paper. We hope our rebuttal addresses your questions and we are happy to answer any further questions you may have.\n\n[1] Pei et al., Graph Alignment with Noisy Supervision, WWW 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The noise problem solved in this paper is important for Weakly-Supervised learning, which is a common dilemma when applying pseudo label generation.\n2. This article is well organized and is well illustrated to help the reader understand.\n3. The experiments designed in this paper are sufficient, not only to design ablation experiments for exploring different components of the model, but also to explore the change of pseudo label quality during the iteration process.\n4. The idea of using two PLM fine-tuning methods for co-training is interesting, and the performance improvements are also obvious.\n", "reasons_to_reject": "1.The reasons for some detail design in the method are not well explained.", "questions_to_authors": "A. In the prompting-based fine-tuning phase, the dataset is split into multiple subsets to train multiple classifiers respectively. With little training sample, is it possible to cause overfitting of a single classifier and reduce its robustness?\nB. Why is the threshold t_i set to increase linearly?\nC. Why does the ELECTRA+ELECTRA method perform so poorly in the YELP dataset in Table 5?\nD. It seems that the benefits of the proposed pseudo label generation method are not analyzed in the experiment?\n", "paper_topic_and_main_contributions": "For weakly supervised text classification, this paper proposes to apply the zero-shot prompting of PLM to obtain pseudo labels. To alleviate the noise problem induced by the pseudo label generation, this paper designs an iterative ensemble training method, which utilizes two PLM fine-tuning strategies and a model ensemble strategy to improve the quality of the pseudo label.  Experimental results show that the proposed method can achieve SOTA on seven datasets, and the performance is close to that of the fully-supervised baseline in two datasets.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers are cautiously positive in their assessment of the paper (3-4).  The proposed method of quickly and cheaply creating labels can be applied to other tasks outside of those tested in this paper.   The agreed upon weakness of the paper is an unclear methodological description, which can be resolved in revisions.  The authors made an effort to address concerns by reviewers.
Decision: Accept-Main