Boosting Prompt-Based Self-Training With Mapping-Free Automatic Verbalizer for Multi-Class Classification
Authors: Yookyung Kho
Jaehee Kim
Pilsung Kang
Venue: EMNLP 2023 Findings (2023)
Keywords: Prompt-based Learning
Prompt-based fine-tuning
Prompt-based self-training
Verbalizer
Label Word Mapping
TL;DR: Prompt-based self-training for multi-class text classification is under-explored, and a new structure called MAV shows promising self-training performances by automatically extracting word features from MLM predictions.

Abstract: Recently, prompt-based fine-tuning has garnered considerable interest as a core technique for few-shot text classification task. This approach reformulates the fine-tuning objective to align with the Masked Language Modeling (MLM) objective. Leveraging unlabeled data, prompt-based self-training has shown greater effectiveness in binary and three-class classification. However, prompt-based self-training for multi-class classification has not been adequately investigated, despite its significant applicability to real-world scenarios. Moreover, extending current methods to multi-class classification suffers from the verbalizer that extracts the predicted value of manually pre-defined single label word for each class from MLM predictions. Consequently, we introduce a novel, efficient verbalizer structure, named Mapping-free Automatic Verbalizer (MAV). Comprising two fully connected layers, MAV serves as a trainable verbalizer that automatically extracts the requisite word features for classification by capitalizing on all available information from MLM predictions. Experimental results on five multi-class classification datasets indicate MAV's superior self-training efficacy.

Boosting Prompt-Based Self-Training With Mapping-Free Automatic Verbalizer for Multi-Class Classification
Yookyung Kho, Jaehee Kim, Pilsung Kang
Korea University, Seoul, Republic of Korea (yookyung_kho, jaehee_kim, kalssung_kaggJ@koree ac kr.
Abstract
et al. (2021a) introduced novel prompt-based fine-tuning methodology. known as PET and LM- BFF which applied prompting to encoder-only models such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019). Their research demon- strated the efficacy of prompt-based fine-tuning in few-shot text classification tasks, as it significantly outperformed standard fine-tuning methods.
Recently, prompt-based fine-tuning has gar- nered considerable interest core technique for few-shot text classification task. This ap- proach reformulates the fine-tuming objective to align with the Masked Language Modeling (MLM) objective. Leveraging unlabeled data, prompt-hased self-training has shown greater effectiveness in binary and three-class classi- fication. However, prompt-bass based self-training for multi-class classification has not been ade- quately investigated, despite its significant ap- plicability to real-world scenarios. Moreover, extending current methods to multi-class classi- fication suffers from the verbalizer that extracts the predicted value of manually pre-defined sin- gle label word for each class from MLM pre- dictions. Consequently, we introduce novel, efficient verbalizer structure, named Mapping- free Automatic Verbalizer (MAV). Compris- ing two fully connected layers, MAV serves trainable verbalizer that automatically ex- tracts the requisite word features for classifica- tion by capitalizing on all available information from MLM predictions. Experimental results on five multi-class classification datasets indi- cate MAV's superior self-training efficacy.
To address the scareity of labeled data, semi- supervised learning approaches have also been ex- tensively researched by leveraging unlabeled data, which often contain sufficient contextual informa- tion (Xie et al., 2020: Chen et al., 2020; Li et al., 2021). Recent studies (Schick and Schütze, 2021; Chen et al., 2021: Zhao and Yao, 2022; Wang et al., 2022b) have experimentally demonstrated the efficacy combining prompt-based fine-tuning with self telfntrainin.. However, existing research on prompt-based self-training has focused on address- ing binary classification (e.g.. Sentiment Analy- sis) or three-class classification (e.g.. Natural Lan- guage Inference) tasks (Chen et al., 2021; Wang et al., 2022b). Although leveraging unlabeled data has been crucial aspect of multi-cla classifica- tion (Song al., 2011; Li et al., 2019; Tang et al., 2022), prompt-based self-training approaches for multi-class classification remain under-explored.
1 Introduction
The language model has demonstrated impressive results numerous practical applications by under- going extensive pre-training using objectives such as masked language modeling and autoregressive language modeling (Peters et al., 2018; Radford and Narasimhan, 2018; Devlin et al., 2019). Re- cently, the prompting approach, as proposed by Radford et al. (2019) and Brown et al. (2020), has played a critical role in addressing the few- shot scenario in various natural language process- ing tasks (Chen et al., 2022: Ma et al., 2022: Liu et al., 2023). Schick and Schütze (2021) and Gao
The challenge in applying conventional prompt- based self-training approaches to multi-class clas- sification arises from the verbalizer. In prompt- based fine-tuning, given input sentence (c.g., "I am happy.") is wrapped with pre-defined nat- ural language sequence including [MASK] token, called template (e.g.. "x: is [MASK].") Subse- quently, verbalizer, a mapping from a label word (e.g., "good") to specific class (e.g., positive), ex- tracts predicted value of the label word from the MLM prediction to generate the final predictive probability distribution. That is. the probability
'Our code is publicly available at yookyungkho/MWV.
Patterr Exploiting Training Better Few-shot Fine-tuning of Language Models

---

that the label word is predicted at the masked posi- tion is regarded as the prediction probability of the class.
prompts (Schick and Schüitze, 2021), Various stud- ics have aimed to answer to critical research ques- tion regarding verbalizer, which significantly in- fluences prompt-based learning performance (Gao et al., 2021a; Webson and Pavlick, 2022): How to identify suitable label words for given class?
The main problem is that most of self-training studies construct verbalizers using manually se- lected single label words. Creating such a man- ual verbalizer necessitates domain knowledge (Gao et al., 202 which causes the high cost asso- ciated with manually selecting label words for numerous classes under the constraint of choos- ing a label word as a single token. Furthermore. extracting only a small amount of limited infor- mation corresponding to label words from high- dimensional MLM predictions can result in infor- mation loss (Hu et al., 2022), leading consid- erable performance disparities depending the selected label word (Gao al., 2021a; Webson and Pavlick, 2022).
Manual verbalizer Schick and Schütze (2021) manually chose a single label word for each class. Selecting a single label word can encompass vari- ous heuristic methods, including utilizing the class name itself or relying on an expert with specialized knowledge.
Automatic verbalizer Gao al. (2021a) high- lighted the sub-optimality of manual approach and proposed an automatic search method to determine the optimal single label word combination by eval- uating candidates generated through MLM infer- ence. Shin et al. (2020) and Schick et al. (2020) introduced multiple label words search strategy based on gradient and loss. Wang et al. (2022a) performed MLM inference before fine-tuning and designated the top-k tokens as label words. Addi- tionally, Hu et al. (2022) proposed a hybrid man- ual and automatic verbalizer that expands the la- bel word space by leveraging external knowledge bases.
To address the limitations of current verbalizers, we propose a novel Mapping-free Automatic Ver- balizer (MAV) that consists of two fully connected layers (FCLs) to automatically extract the neces- sary vocabulary information for classification by leveraging entire MLM predictions without speci- fying explicit label words. The proposed method- ology MAV has the following advantages: (1) eliminates the need for manual selection of label words. (2) leverages the entire MLM predictions without information loss. (3) It circumvents the cost issue associated with searching for optimal label words. (4) It can be applied to any multi-class datasets, irrespective of the number of classes.
Verbalizer-free methods The cost of label engi- neering main challenge in prompt- fine- tuning. Thus, verbalizer-free methodologics em- phasize the importance of exploiting the [MASK] representation rather than the MLM prediction it- self (Cui et al., 2022; Karimi Mahabadi et al., 2022; Xu et al., 2023). These approaches rely solely on the [MASK] representation and employ distance- based learning to bring instances of the same class closer in the representation space.
In this study, we conducted few-shot text clas- sification on five multi-class datasets, achieving an average performance improvement of 12.8% over existing self self-training methodology. Quantita- tive metrics demonstrated that the proposed MAV benefits most from self eefftrrainng. Further analysis revealed that MAV is proficient in extracting vocab- ulary features representative of each class, thereby enhancing the efficacy of prompt-based learning without manual manipulation of the verbalizer.
2.2 Prompt-based self-training
Contrary to widely studied prompt-based fine- tuning, prompt-based self-training has received relatively less attention. Chen et al. (2021) intro- duced SFLM*, a method combining manual label words from Gao et al. (2021a) with lseudo-labeling and consistency regularization algorithms from Fix- Match (Sohn al., 2020) for self-training. Zhao and Yao (2022) incorporated the consistency reg- ularization loss term from Xie et al. (2020) into the self-training process of SFLM. Wang et al.
2
Related Work
2.1 Prompt-based fine-tuning
Prompt-based fine-tuning reformulates the fine- tuning objective MLM objective. This approach ensures that pre-trained knowledge is fully lever- aged the fine-tuning process (Gao et al., 2021 Han et al., 2022; Liu et al., 2023), while also incorporating spec-fc information through
"Self-training techniques and a data-cfficient Few-shot learner of Language Model

---

(a) Single Label Word Mapping
[M] Where the Octocco River?
Lacader
M
PLM
MLM Head
Eucottin of Location)
FCL MUAA
Decoder PP)
Word Embedding
RoBERTa
Multi Label Words Mapping
M
Sum
Yaoon LLoution
Laattn
Freese
Mapping-frest Automatic Verbalizer (MAV)
MLM Head
PCL
Decoder 19
Vocab Extractor (V],,,))
Output Layer 61,. sl
(6,
Figure 1: Illustration of MAV and comparison with other verbalizers. (a), (b): The input sequence with the template added is fed into the PLM and MIM Head, which returns prediction vector of [MASK] token. Among MI.M prediction, logit values of one or multiple pre-defined label words of each class are extracted form final prediction vector by mapping function M. (c): MLM prediction is calculated while the parameters of the MLM Head are frozen. MAV feeds MLM prediction into two FCLs to return the final probability distribution, automatically identifying the necessary vocabulary information for class distinction.
(2022b) conducted self-training with a teacher- mudent-based adapter tuning method. Most of these studies target binary and three-class classifi- cation. Furthermore, they all rely the manual single-label word mapping of Schick and Schütze (2021) and Gao et al. (2021a), potentially causing information loss and sub-optimal performance.
a pre-defined template as shown in Eq. (1). In this study, we used the manual template (*[MASK] :") from Gao et al. (2021a).
= [CLS] [MASK] x [SEP]:
(1)
The MLM prediction of the [MASK] token R|V generated by the Pre-trained Language Model (PLM) and the MLM head.
Furthermore, it is challenging to integrate the verbalizer enhancement methodologies mentioned in Section 2.1 with self-training for multi-class classification. Specifically, automatic verbalizer approaches require additional data for label word search and incur significant computational burden when the number of classes increases. Moreover, verbalizer-free methods rely on spatial distances, making it impossible to generate explicit predic- tive probability distributions essential for the self- training process. In contrast, proposed MAV is specialized verbalization method for multi-class classification that offers cost-saving benefits, eliminates the need for label word search. More- over, MAV utilizes all MLM prediction informa- tion, allowing pre-trained knowledge to be fully leveraged in downstream tasks.
Existing verbalizers (Figure (a) and (b)) con- struct the final predictive probability distribution with the MLM prediction
P (g/xx) - (M = u|x')/u E Vg), (2)
where Vy, is set of pre-defined label words of specific class y, and a function M (e.g., identity function, sum) transforms predicted values of la- bel words to the probability of the class. These approaches require additional costs for label word selection.
In contrast, shown in Figure 1 (c), Mapping- free automatic verbalizer (MAV) consists of two FCLs and feeds the MLM prediction v
P. (vlx) softmax (W.- Tanh (W Tanh (v))).
3 Methodology
(3)
W E R Vixdee is a weight matrix of the first FCL, named Vocab Extractor (VE). It extracts use- ful information from high dimensional ((V) vocab- ulary features to low dimensional (dve) representa- tion. dye is set to 256 via hyper-parandn tuning
3.1 Mapping-free automatic verbalizer
The procedure of prompt-based fine-tuning is shown in Figure 1. For a multi-class classification task, the given input sequence x is augmented with

---

Real Label
Labeled Sample
Sou
[MASK]: : Nome female Tigure skaler
PL
Hood
MAV
Peeudo Label
Weak--uug ug (Dmooott)
Unlabeled Sample
[MA SK is the capital Kosovo?
PLM
[MASK]: What the capital of Kosovo
Streeg-Aug [Random M Masking)
[MASK]: What the [MASK] Kosovo
PLMM
Last
Random Masking
[MASK] : W hat the capital of [MASK]
?
PLMM
MLM Head
Catton
Figure 2: Overall Self-training Framework. The labeled sample with the template is processed through the prompt framework including MAV to return the predictive probability distribution, and the supervised loss is calculated with the ground-truth label. The unlabeled sample weakly augmented generate pseudo-label. The self-training loss is computed with the pseudo label and the predicted probability distribution derived by strongly augmented sample. Furthermore, the auxiliary MLM loss calculated by applying random masking on the unlabeled sample. Note that [MASK] highlighted in red represents the token used in MLM prediction.
(see Appendix B.2). The subsequent FCL with a weight matrix Wa E RRivv Xc generates the final pre- dictive probability distribution for each class based on the compressed information. This approach con- verts the verbalizer into learnable form without providing any label word information related to the class. Therefore, MAV autonomously identify the vocabulary information needed distinguish between classes in the MLM prediction.
beled samples in the batch, respectively. is the number of labeled samples the batch, and is the ratio of labeled and unlabeled samples. Each loss is computed as follows.
(i (1, B)), (1 (1, BB)
(5)
Supervised loss For a labeled sample x,, the pre- dictive distribution y; = P (y) x²) is obtained from the MAV framework. The Cross Entropy (CE) loss is measured between y; and the ground truth label Y; expressed one-hot vector.
Another significant consideration in construct- ing MAV is the fixed parameters of the MLM head. During the pre-training phase, the decoder of the MLM head shares parameters with the word em- bedding matrix, meaning it contains pre-trained vocabulary representation information. In order to preserve this information and reduce training cost, the MLM head is frozen during the fine-tuning phase. The effect of freezing parameters dis- cussed in Section 4.5.
B Σ 1=1
Suup
B CE (v).
(6)
Self-training loss Lat, a component of the un- supervised loss, aims to perform consistency reg- ularization by taking weak and strong augmenta- tion to the unlabeled sample that the predic- tive distribution of the strongly augmented sample closely matches the pseudo-label generated from the weakly augmented sample.
3.2 Overall training framework
The overall self-training framework using MAV minimizes supervised and unsupervised losses calculated based on labeled and unlabeled data, depicted in Figure 2. The unsupervised loss comprises the self-training loss based Fix- Match (Sohn al., 2020) and the auxiliary MLM loss, following the previous research, SFLM (Chen et al., 2021), We construct the final loss term with weighting factors, X1 and λz, as
BB
Lst =
Σn
max
27)
BB
(7)
CE
wwakk mostong
where q: queak and q;- getrong denote the predictive dis- tributions of the weakly and strongly augmented samples, respectively. q represents the pscudo label for the weakly augmented sample, taking
Liatal = Loup + X1 Let + Xg Lyntn-
(4)
Let x; and u; in Eq. (5) denote labeled and unla-

---

the form of hard label (one-hot vector). Fol- lowing the thresholding technique of FixMatch, the self-training loss is computed only for samples where the confidence of the weakly augmented sample (max (qgroak) exceeds a pre-determined threshold (7), set to 0.95. Additionally, we imple- mented dropout and random masking weak and strong augmentation methods, following Chen et al. (2021) (see Appendix B.2 for details).
Class (Before)
Class (After)
Dataset
Type
TREC
Topic
6
Trec50
Topic
50
22
GoEmotions
Emotion
28
26
Yahoo Answers
Topic
10
10
16
AG's News
Topic
4
4
16
Table Data description. Each column indicates the name of dataset, the type of classification, the number of classes before and after data pre-processing. the num- ber of labeled data per class (k), and the ratio between labeled and unlabeled data size (u).
Auxiliary MLM loss Lmim is an auxiliary loss proposed by Schick and Schütze (2021) to prevent catastrophic forgetting often observed when fine- tuning with small amount of labeled data. Chen al. (2021) added similar self-supervised loss that predicts the [MASK] token of the strongly aug- mented sample for regularization. In this study, we adopted the auxiliary MLM loss of PET and applied 15% random masking to the unlabeled sam- plc, separate from the strong augmentation.
TREC50, and GoEmotioins dataset, some classes did not fulfill the minimum data size per class. such cases, we maintained a constant ratio of unla- beled data (μ) but decreased the value of k. which means fewer data for each class. Some classes that did not meet the minimum data requirement were excluded from the experiment. Details of datasets are provided in Table and Appendix B.I.
pB
1
mmlmm
mime
B.B
4.1.2 Baselines
1=1
Three baselines were employed to evaluate the per- formance of prompt-based self-training with our proposed MAV.
where Cimm denotes the average MLM loss of masked tokens (except the one in the template) in the jth sequence:
Standard fine-tuning is a traditional fine-tuning technique that does not utilize templates verbal- izers. Instead, it performs classification using the [CLS] token representation by adding classifica- tion head after the PLM.
log P ([MASK]] = RM (u()),
m
==1
where m represents the number masked tokens in the 5th sequence, and i denotes the target of the 5th [MASK] token. RM means Random Masking.
Single label word takes only the logit value of single label word mapped to a class in MLM prediction. We adopted the manual label words used in Schick and Schüizee (2021) and Gao et al. (2021a). Implementation details can be found in Appendix B.3.
4 Experiments
4.1
Experimental setup
Multi label words take the logit values of mul- tiple label words mapped to each class in MLM prediction. We adopted AMuLaP (Wang et al., 2022a), a parameter-free automatic verbalizer con- struction method using only small number of la- beled samples, the Multi Label Words baseline. Details can be found in Appendix B.3.
4.1.1 Dataset
We evaluated MAV on five multi-class datasets; TREC (Hovy et al., 2001), TREC50 (Li and Roth, 2002), GoEmotions (Demszky al., 2020), Ya- hoo Answers (Zhang et al., 2015), and AG's News (Zhang et al., 2015). The datasets encom- pass minimum of four classes and extend up to dozens of classes, thereby confirming versatility across datasets with different numbers of classes. Following the experimental settings of Chen et al. (2021), we set the number of labeled data per class = 16) and unlabeled data ratio (μ 4) per class, resulting in a minimum training data size of 80 (16 + 16 4) for each class. However, for TREC,
Furthermore, we explored the verbalizer free approach discussed in Section as a baseline. However, this methodology is not applicable to the self-training approach employed in this study. Therefore, we conducted an alternative experiment, which described in Appendix C.
AAttomatic Multi-Label Prompting

---

TREC
TREC50
GoEmotions
Yahoo
AG's News
Small-supervised
70.1 (4.6)
80.0 (6.5)
21.2 (1.6)
63.4 (1.8
84.6 (0.9)
Standard Fine-tuning
Semi-supervised
80.2 (4.2)
79.4 (9.4)
21.9 (1.5)
64.4 (1.4)
86.3 (1.1)
Full-supervised
90.8 (3.1)
91.2 (2.0)
36.4 (1.0)
68.7 (0.3)
88.5 (0.8)
Benefit Ratio
0.49
-0.05
0.05
0.19
0.44
Small-supervised
77.2 (6.3)
84.4 (6.0)
15.7 (2.2)
65.5 (1.5)
86.1 (1.0)
Single Label Word
Semi-supervised"
81.4 (3.1)
81.5 (12.1)
16.8 (1.0)
68.0 (0.7)
87.3 (0.8)
Full-supervised
91.6 (1.0)
93.1 (1.5)
32.8 (2.0)
69.2 (0.9)
88.1 (0.6)
Benefit Ratio f
0.29
-0.33
0.06
0.66
0.60
Small-supervised * Semi-supervised Full-supervised
72.7 (6.0)
64.6 (1.2)
17.3 (1.6)
57.6 (2.2)
81.4 (1.5)
(2.3)
82.0 (2.1)
18.0 (1.3)
61.1 (1.7)
84.1 (0.7)
Multi Label Words
81.4 (3.1)
89.7 (3.3)
32.8 (1.4)
68.2 (0.5)
88.1 (0.8)
Benefit Ratio
0.21
0.69
0.05
0.33
0.40
Small-supervisee
77.1 (9.9)
(5.2)
24.2 (1.7)
64.7 (2.4)
84.3 (2.2)
Semi-supervised
85,4 (3.7)
87.9 (5.1)
25,4 (2.5)
68.1 (0.9)
87.2 (0.9)
MAV (ours)
Full-supervised
93.9 (1.1)
91.3 (2.0)
36.1 (1.2)
69.4 (0.5)
89.0 (0.5)
Benefit Ratio f
0.50
0.74
0.10
0.73
0.63
Table 2:
Results of few-shot multi-class classification. Each cell is filled with the average accuracy of the five random seeds, with numbers brackets indicating the standard deviation. For each methodology, we report performance of small-supervised, semi-superrvieed, and full-supervised model, along with benefit ratio to measure the effectiveness of self-training, Small-supervised and full-supervised models are trained with the supervised loss outlined in Section 3.2. while semi-supervised model trained with both self-training loss and auxiliary MLM loss in addition to the supervised loss. The highest self aiff-traning performance and benefit ratio achieved in each dataset are emphasized in bold text. The superscript symbol denotes previous rescarh corresponding to each setting FixMatch (Sohn et al., 2020), LM-BFF (Gao al., 2021a). SFLM (Chen al.. 2021), AMuLaP (Wang et al., 2022a)).
4.1.3 Experimental details
The performances of small-supervised and full- supervised models indicate the lower and upper bounds for the semi-supervised model. In other words, small-supervised performance showcases how semi-supervised model benefits from incor- porating unlabeled data with limited labeled data, while full-supervised performance represents the upper limit in the ceni-supervised context. We calculated the benefit ratio (Zhu et al., 2022) as
PLM We adopted RoBERTa-bast (Liu et al., 2019) as the PLM for all experiments.
Evaluation Following Gao et al. (2021a), we sampled training and validation data under five random seeds and evaluated them using the orig- inal test dataset, measuring the average accuracy and standard deviation. To adhere the few-shot text classification assumption, we set the size of the validation dataset be the same as that of the training dataset. Furthermore, hyper-parameters are tuned based on validation performance. Details of hyper-parameter selection are in Appendix B.2.
Acc (Semi) Acc (Small) Acc (Full) Acc (Small) (10)
Benefit Ratio
measuring the performance improvement of semi- supervised model compared to small and full super- vised model. The closer the benefit ratio is 1, the more comparable the performance of the semi- supervised model is to the full-supervised model.
Quantitative metric To assess self-training ef- fectiveness, we measured the performance of three settings as follows: Small-supervised model trained with a small amount of labeled data (k la- beled samples per class), Semi-superviser model trained with small amount of labeled data and μ times unlabeled data (k labeled and uk unlabeld samples per class), and Full-supervised model trained with full labeled data (k + fk labeled sam- ples per class).
4.2 Experimental results
Table 2 presents the experimental results for multi- class classification. The proposed MAV achieves
"The benefit ratio was originally proposed to measure the learning difficulty of each class. In this paper, we aggregated results from all classes to demonstrate the effectiveness of semi-supervised learning.

---

standard fine-tuning. Sihoustte soore: 0.3989
Sihoustt soors: 0,5903
Sihoustte score: 0, 4219
Sihoustte soore: 0.6799
(a) Zero-shot Inference
(b) Single Label Word
(c) Multi Label Words
(d) MAV (ours)
Figure 3: t-SNE visualization of [MASK] representations from baselines and MAV. (a) Zero-shot inference before fine-tuning, (b) Single Label Word Mapping. (c) Multi Label Words Mapping, (d) Mapping-free Auto Verbalizer results. The numbers below each plot represent Silhouctte scores. We used TREC dataset, which contains six classes, and randomly sampled one of the five seeds.
the highest self-training performance across all baselines, except AG's News dataset, with an av- erage self-training performance improvement of 12.8% over the existing prompt-based self-training methodology, SFLM (Single Label Word, Semi- supervised). Moreover, MAV attains the highest benefit ratio for all datasets, demonstrating that is the most advantageous verbalizer structure for self-training in multi-elass classification tasks. In contrast, Single Label Word does not fully bene- fit from self- self-training, despite performing well in setting. Moreover, Single Label Word requires additional costs to manually specify the label word for every class.
Overall, the proposed MAV maximizes the ef- fectiveness of prompt-based self- seff-training by fully leveraging MLM prediction without any additional cost for label word engineering.
4.3 Qualitative analysis
The effectiveness of prompt-based self-training in text classification, which utilizes MLM predictions from [MASK] representations, is heavily influ- enced by the degree of aggregation of these repre- sentations by class (Cui al., 2022). Therefore, we examined the aggregation of [MASK] represen- tations by class within the test dataset using t-SNE plot and Silhouette scores. As shown in Figure 3, MAV (Figure 3 (d)) forms much denser clusters of [MASK] representations compared to zero-shot inference (Figure 3 (a)) and explicit label word mapping methodologies (Figure 3 (b) and (c)). Ad- ditionally, applied K-Means clustering with the number of clusters equal to the number of classes and calculated the silhouette score. Both i-SNE plots and Silhouette scores demonstrate that the clustering results of MAV are superior to the base- lines. This suggests that leveraging all information from MLM predictions is beneficial for multi-class classification in prompt-based self-training than using limited information from MLM predictions.
The self-training performance MAV and Sin- gle Label Word baseline is comparable for AG's News, which has fewer classes than other datasets, and for Yahoo Answers, which has a relatively clear distinction between classes. However, MAV offers clear advantage in that it does not require manual intervention on the label words and can be scaled to other datasets regardless of the number classes. For the GoEmotions dataset, which in- volves categorizing over 20 fine-grained emotions, the classification task is challenging, and the impact of self-training is uncertain across all approaches due to relatively weak classifier. Nonetheless, MAV significantly outperforms Single Label Word and Multi Label Words, which utilize limited infor- mation when class distinctions are unclear.
4.4 Comparison of verbalizers
Interestingly, our findings contradict the com- mon belief that prompt-based fine-tuning always outperforms standard fine-tuning in few-shot set- tings (Gao al., 2021a; a; Chen al., 2021). Our results indicate that standard fine-tuning is not sig- nificantly inferior in multi-class classification tasks. Moreover, a considerable drop performance of
The core motivation of MAV is to construct learn- able verbalizer that enables the model to identify the vocabulary features helpful for classification. This section investigates whether MAV focuses on the most appropriate vocabularies that represent the unique characteristics of each class. Since the verbalizer of Multi Label Words (Table (b)) was built with only a small amount of labeled data, we
Multi Label Words is observed when compared to

---

Class
(4)
Single Label Word
(b) Multi Label Words
(c) MAV (ours)
Poll. Related. RELATED, More,
Map, city, Map². Maps, map, Location, cities*, map*. Paris.
Location
Location
MORE, Video. Query, Bonus....
Next, Note, NOTE, UPDATE, Quick, Advertisement, Edit, Trump, Now,
Who*, Winners", Name, winners*, named*, Guests", nominated",...
Human being
Human
Question, Analysis, Myth, AP, Correction, GM, CNN, BP, Claim,
1985, Time, RM, 1968, 1969, 1982, 1965, 1984, Temperature, 1978
Numeric value
Number
Table 3: Comparison of verbalizers. The first column involves three randomly sampled classes in TREC dataset, while each cell of the other three columns indicates (a) single label word mapped to each class, (b) automatically explored multiple label words through LMML for each class, and (c) vocabulary features that contributed the most to the prediction with each class. We randomly sampled one of the five seeds for (b) and (c). Note that the word with * (e.g., 'Gacronym') ") refers to a token with a space symbol.
validated the capability of MAV under more chal- lenging amall-supervised setting without the benefit of unlabeled data. SHapley Additive exPlanations (SHAP), proposed by Lundberg and Lee (2017), was utilized to identify the words that contributed the most to the final prediction.
Parameter freeze
TREC
TREC50
GoEmotions
No MLM Head (ours)
85.7 (4.1)
87.4 (2.9)
25.3 (1.6)
85.4 (3.7)
87.9 (5.1)
25,4 (2.5)
RoBERTa
50.1 (8.6)
36.1 (8.4)
13.9 (1.7)
Table 4: Self-training performances MAV depending on freezing parameters. The three settings are as follows: No (updating all parameters), MLM Head (freczing only the MLM head), and RoBERTa (freezing the RoBERTa encoder).
Specifically, we calculated the SHAP value for each class using only the samples that answered correctly in the test dataset and listed top features (tokens), as shown in Table 3 (c). The empirical evidence supports that MAV truly has the ability to utilize words (e.g.. 'Map', "city', 'Location)) ") as- sociated with the class (e.g., 'Location') ') for multi- class classification. This is remarkable since MAV does not refer to any label word information about the class at all. On the other hand, for Multi La- bel Words, the lowest performer, selecting label words through zero-shot MLM inference with only a small amount of labeled data is unsuitable for multi-class classification tasks where it is relatively difficult to distinguish between classes.
TREC
TREC50
GoEmotions
MAV (ours)
83.6 (5.7)
87.0 (3.6)
25.4 (1.4)
Single Label Word
82.6 (3.5)
83.0 (8.2)
15.2 (0.8)
Table 5: Self-training performances of FlexMatch.
of the fully fine-tuned case. At the same time, it significantly outperforms the scenario where the RoBERTa encoder is frozen. This suggests that the MLM head is sufficiently adept at generating MLM predictions that earry valuable information about the downstream data even without further tuning.
4.5 Effect of freeziig parameters
4.6 Effect of self-training methods
An important feature of MAV is that it freezes the parameters of the MLM head that produces vo- cabulary logits. This freezing mechanism aims to generate informative predictions that are not biased by the small amount of labeled data during the fine-tuning process. Simultaneously, we harnessed the pre-trained knowledge embedded within the decoder of the MLM head. We compared the per- formance of prompt-based self-training based on the MAV structure with different parameter update scenarios while following the overall self-training framework Figure 2. According to Table 4, self- training performances of freezing the parameters of the MLM head are almost identical to the results
We adopted FixMatch as the self-training method- ology for fair comparision with SFLM. To verify the robustness of MAV when implementing dif- ferent self-training algorithms besides FixMatch, we conducted a comparative analysis using Single Label Word, which demonstrated the best perfor- mance among the baseline models. Our experi- ments included FlexMatch (Zhang et al., 2021), improved self-training algorithm that adjusts the threshold based on the difficulty of each class. The results are illustrated in Table 5. The performance gaps between MAV and Single Label Word are consistent across the self-training methods.

---

5 Conclusion
following previous research (Gao et al., 2021a), five sub-datasets were randomly sampled and uti- lized in the experiments. To ensure reproducibility and facilitate future research on this topic, we re- lease all resources including datasets and code, and appropriate citations to previous research are indi- cated within the code.
In this research, we proposed a simple and effec- tive verbalizing method, Mapping-free Automatic Verbalizer, to enhance prompt-basee self- training performance on various multi-class classification tasks. Unlike previous verbalizers which take lim- ited information from MLM prediction and require label engineering, MAV is trainable verbalizer consisting of two FCLs that extracts key vocab- ulary features from MLM predictions and maps them to label space. This approach has the advan- tage of promoting the efficiency of prompt-based learning without manual verbalizer manipulation or the search for optimal label words. Experimental findings not only validate the effectiveness of MAV but also highlight its self-training efficacy across all datasets, supported by quantitative metrics and qualitative analysis.
Furthermore, our research focused investigat- ing news topic and sentiment classification tasks that are not anticipated to have any negative social impacts. However, when it comes to classification tasks that have the potential to raise ethical con- cerns regarding race and gender, it is crucial to adopt more careful and cautious approach while analyzing the results.
References
Armen Aghajanyan, Akshat Shrivastava, Anchit Gupta, Naman Goyal, Luke Zettlemoyer, and Sonal Gupta. 2021. Better fine-tuning by reducing representational collapse. In 9th International Conference on Learn- ing Representations, ICLR 2021, Virtual Event, Aus- tria, May 3-7, 2027. OpenReview.ner.
Limitations
Due to the instability and high sensitivity of few- shot learning to hyper-paramede selection, rel- atively large standard deviation can be seen in certain cases in Table 2 (Semi-supervised model with Standard Fine-tuning for TREC50, Semi- supervised model with Single Label Word for TREC50, Small- supervisd model with MAV for TREC). This is mainly caused by data sampling and hyper-parameter tuning being performed separately for each seed and the small size of the validation data. Additionally, the checkpoint with the highest validation performance may not necessarily yield the best results on test dataset, resulting in signif- icant variance. These are inherent limitations of few-shot learning. To provide clarity, we provide the performance by seed for these three cases in Appendix A.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler. Jeffrey Wu. Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates. Inc.
Jiaao Chen, Zichao Yang, and Diyi Yang. 2020. Mix- Text: Linguistically-infrmeed interpolation of hid- den space for semi-supervised text classification. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 2147- 2157, Online. Association for Computational Lin- guistics.
In this study, we did not address scenarios where the class distribution of the dataset is im- balanced. It will be worth investigating a prompt- based self-training approach that can handle im- balanced datasets, which is significant challenge in real word applications. Furthermore, we sug- gest exploring self-training methodologies tailored to the prompt framework that align with MLM objectives, moving beyond the application of self- training methodologies from the vision field.
Xiang Chen, Lei Li, Ningyu Zhang, Chuanqi Tan, Fei Huang, Luo Si, and Huajun Chen. 2022. Rela- tion extraction open-book examination: Retrieval- enhanced prompt tuning. In Proceedingy of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '22, page 2443-2448, New York, NY, USA. Associa- tion for Computing Machinery.
Ethics Statement
Yiming Chen, Yan Zhang, Chen Zhang, Grandee Lee, Ran Cheng, and Haizhou Li. 2021. Revisiting self- training for few-shot learning of language model.
We honor the ACL Code of Ethics. All datasets employed in this study are publicly available, and

---

In Proceedings of the 2021 Conference on Empir- ical Methods in Natural Language Processing, pages 91 9125-9135, Online and Punta Cana, Dominican Re- public. Association for Computational Linguisties.
Shengding Hu, Ning Ding, Huadong Wang. Zhiyuan Liu, Jingang Wang, Juanzi Li, Wei Wu, and Maosong Sun. 2022. Knowledgeable prompt-tuning: Incor- porating knowledge into prompt verbalizer for text classification. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 2225-2240, Dublin, Ireland. Association for Computational Lin- guistics.
Ganqu Cui, Shengding Hu, Ning Ding, Longtao Huang, and Zhiyuan Liu. 2022. Prototypical verbalizer for prompt-based few-shot tuning. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Pupers). pages 7014-7024, Dublin, Ireland. Association for Compu- tational Linguistics.
Rabeeh Karimi Mahabadi, Luke Zettlemoyer, James Henderson, Lambert Mathias, Marzieh Saeidi, Veselin Stoyanov, and Majid Yazdani. 2022. Prompt- free and efficient few-shot learning with language models. In Proceedings of the 60th Anmual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5638-3652, Dublin, Ireland. Association for Computational Linguistics.
Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. 2020. GoEmotions: A dataset of fine- ane-grained emo- tions. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4040-4054, Online, Association for Computational Linguistics.
Jaehyung Kim, Dongyeop Kang, Sungsoo Ahn, and Jinwoo Shin. 2022. What makes better augmentation strategies? augment difficult but not too different. In The Tenth International Conference on Learning Representations, ICLR 2022, Viriual Event, April 25- 29, 2022. OpenReview.net.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Changchun Li, Ximing Li, and Jihong Ouyang, 2021. Semi-supervised text classification with balanced deep representation distributions. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference Natural Language Processing (Volume 1: Long Papers), pages 5044-5053, Online, Association for Computational Linguistics.
Steven Y. Feng, Varun Gangal, Jason Wei, Sarath Chan- dar. Soroush Vosoughi, Teruko Mitamura, and Ed- uard Hovy. 2021. A survey of data augmentation approaches for NLP. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 968-988, Online. Association for Computa- tional Linguisties.
Jian Li, Yong Liu, Rong Yin, and Weiping Wang. 2019. Multi-class learning using unlabeled samples: The- ory and algorithm. In Proceedings of the 28th Inter- national Joint Conference on Artificial Intelligence, IJCAI' 19, page 2880-2886. AAAI Press.
Tianyu Gao, Adam Fisch, and Danqi Chen. 2021a Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers), pages 38 16-3830, Online. Association for Computa- tional Linguisties.
Xin Li and Dan Roth. 2002. Learning question classi- fiers. In Proceedings of the 19th International Con- ference on Computational Linguistics- Volume 1, COLING '02, page 1-7, USA. Association for Com- putational Linguistics.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021b. SimCSE: Simple contrastive learning of sentence beddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 6894-6910, Online and Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Comput. Surv. 55(9).
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy. Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1997. 1169211.
Xu Han, Weilin Zhao, Ning Ding, Zhiyuan Liu, and Maosong Sun. 2022. Ptr: Prompt tuning with rules for text classification. Al Open, 3:182-192.
Eduard Hovy. Laurie Gerber, Ulf Hermjakob. Chin- Yew Lin, and Deepak Ravichandran. 2001. Toward semantics-based answer pinpointing. In Proceed- ings of the First International Conference Human Language Technology Research, HLT '01, page 1-7, USA. Association for Computational Linguistics.
Scott M. Lundberg and Su-In Lec. 2017. A unified approach to interpreting model predictions. In Pro- ceedings of the 31st International Conference on Neu- ral Information Processing Systems, NIPS' 17, page 4768 4777, Red Hook, NY, USA. Curran Associates Inc.

---

Ruotian Ma, Xin Zhou, Tao Gui. Yiding Tan, Linyang Li, Qi Zhang, and Xuanjing Huang. 2022. Temmllte- free prompt luning for few-shot NER. In Proceedings of the 2022 Conference of the North American Chap- ler of the Association for Computational Linguistics: Human Langwage Technologies, pages 5721-5732, Seattle, United States. Association for Computational Linguistics.
Enmin Song, Dongshan Huang, Guangzhi Ma, and Chin-Cheng Hung. 2011. Semi-supervised multi- class adaboost by exploiting unlabeled data. Expert Systems with Applications, 38()):670-66726.
Yuting Tang, Nan Lu, Tianyi Zhang, and Masashi Sugiyama. 2022. Multi-class classification from mul- tiple unlabeled datasets with partial risk regulariza- tion. In Proceedings of the 14th Asian Conference on Machine Learning (ACML2022), Proceedings of Machine Learning Research, Hyderabad, India.
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lec, and Luke Zettlemoyer. 2018. Deep contextualized word repre- sentations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume (Long Papers), pages 2227-2237, New Orleans, Louisiana, Association for Computa- tional Linguistics.
Han Wang, Canwen Xu, and Julian McAuley. 2022a. Automatic multi-label prompting: Simple and inter- pretable few-shot classification. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 5483-5492, Seattle, United States. Association for Computational Linguistics.
Alec Radford and Karthik Narasimhan. 2018, Im- proving language understanding by generative pre- training.
Yaqing Wang, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Awadallah, and Jianfeng Gao. 2022b. LiST: Lite prompted self-training makes parameter-efficient few-shot learners. In Findings of the Association for Computational Linguisties: NAACL 2022, pages 2262-2281, Seattle, United States. Association for Computational Linguistics.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Timo Schick, Helmut Schmid, and Hinrich Schütze. 2020. Automatically identifying words that can serve as labels for few-shot text classification. In Proceed- Com-
Albert Webson and Ellie Pavlick. 2022. Do prompt- based models really understand the meaning of their prompts? In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 2300-2344, Seattle, United States. Association for Computational Linguisties.
ings of the 28th International Conference putational Linguisties, pages 5569-55777 5578,, Barcelona, Spain (Online). International Committee on Compu- tational Linguistics.
Timo Schick and Hinrich Schütze. 2021. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Asso- ciation for Computational Linguistics: Main Volume, pages 255-269,, Online. Association for Computa- tional Linguistics.
Jason Wei and Kai Zou. 2019. EDA:
Easy data augmen- tation techniques for boosting performance on text classification tasks. In Proceedings of the 2019 Con- ference on Empirical Methods in Natural Language Processing and the 9th International Joint Confer- ence on Natural Language Processing (EMNLP- IJCNL.P), pages 6382-6388, Hong Kong, China. As- sociation for Computational Linguistics.
Dinghan Shen, Mingzhi Zheng, Yelong Shen, Yanru Qu, and Weizhu Chen. 2020. A simple but tough- to-beat data augmentation approach for natural language understanding and generation. CoRR, 0bs/2009.13818v//
Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. 2020. Unsupervised data augmenta- tion for consistency training. In Advances Neural Information Processing Systems, volume 33, pages 6256 6268. Curran Associates, Inc.
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. AutoPrompt: Elic- iting Knowledge from Language Models with Auto- matically Generated Prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4222-4235. Online. Association for Computational Linguistics.
Ziyun Xu, Chengyu Wang, Minghui Qiu, Fuli Luo, Runxin Xu, Songfang Huang, and Jun Huang. 2023. Making pre-trained language models and-to-end few- shot learners with contrastive prompt tuning. In Pro- ceedings of the Sixteenth ACM International Confer- ence on Web Search and Data Mining, WSDM 2023, Singapore, 27 February 2023 3 March 2023, pages 438-446. ACM.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang. Han Zhang, Colin A Raffel, Ekin Dogus Li. 2020. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In Advances in Neural Information Processing Systems, volume 33, pages 596-608. Curran Associates, Inc.
Cubuk, Alexey Kurakin, and
Mingyang Yi, Lu Hou, Lifeng Shang, Xin Jiang, Qun Liu, and Zhi-Ming Ma. 2021. Reweighting aug- mented samples by minimizing the maximal expected loss. In 9th International Conference on Learning

---

Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
A
Inherent Limitations of Few-shot Learning
Bowen 7hang, Yidong Wang, Wenxin Hou, HAO WU, Jindong Wang. Manabu Okumura, and Takahiro Shi- nozaki. 2021. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. In Ad- vances in Neural Information Processing Systems, volume 34, pages 18408 18419, Curran Associates, Inc.
As mentioned in Limitations, we provide the perfor- mance by seed in the cases where relatively large standard deviation can be seen. Three cases are described in Table 6.
B Experimental Details
Xiang Zhang. Junbo Zhao, and Yamn LeCun. 2015. Character-level convolutional networks for text clas- sification. In Proceedings of the 28th International Conference on Neural Information Processing Sys- tems Volume 1, NIPS' 15, page 649-657, Cambridge, MA, USA. MIT Press.
B.1 Datasets
As elaborated in Section 4.1.1, it was essential omit certain classes from TREC50 and GoEmo- tions to maintain a balanced self-training frame- work, considering the minimum data requirement of 80 examples per class. We aimed to ensure the integrity of training, extremely small sample sizes for certain classes could hinder robust model learning and data consistency. Some of the elimi- nated classes consist of only a handful of samples, as low as 4 or 6.
Lei Zhao and Cheng Yao. 2022. EICO: Improving few-shot text classification via explicit and implicit consistency regularization. In Findings of the Asso- ciation for Computational Linguistics: ACL 2022, pages 3582-3587, Dublin, Ireland. Association for Computational Linguistics.
Zhaowei Zhu, Tianyi Luo, and Yang Liu. 2022. The rich get richer: Disparate impact of semi-supervised learning. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event. April 25-29, 2022. OpenReview.net.
Since it is not possible to make predictions for classes that are not included in the training, classes that are removed from the train and dev datasets correspondingly excluded from the test dataset. Regarding class balance, the characteristics of test sets differ across datasets. In the case of Yahoo Answers and AG's News, the test sets are intention- ally designed to maintain a balanced distribution of classes. Conversely, TREC, TREC50, and GoEmo- tions exhibit slass-iballanced test sets, reflective of their original nature.
B.2 Hyper-parameter selection
Hidden dimension of vocab extractor The first FCL of MAV, vocab extractor, designed auto- matically extract the vocabulary features required for classification from high-dimensional(()) vo- cabulary logits and map them into a low- dimensional(dxe)) space. To identify an appropriate dimension for feature extraction, we adopted four comparison groups As shown in Table 7, the effec- tiveness of MAV was greatest when the dimension- ality was 256.
Augmentation methods In this study, we adopted dropout and random masking as weak and strong augmentation techniques for consistency regularization, the same as SFLM (Chen al., 2021). Unlike image data, text characterized by its discrete nature within the language domain, where small variations can cause large semantic changes (Feng et al., 2021). Therefore, to preserve

---

Method
Type
Datset
Seed
Seed 42
Seed 87
100
Avg Acc (SD)
Single Standard FT
Semi-supervised
TREC50
89.3
81.2
86.8
89.6
60.7
81.5 (12.1)
Semi-supervised
TREC50
84.3
80.2
84.8
85.0
62.9
79.4 (9.4)
MAV
Small-suprrvsed
TREC
87.6
71.8
84.0
79.2
77.1 (9.9)
Table 6:
Performance of each seed for cases where standard deviation is greater than 9. Note that Standard FT indicates Standard Fine-tuning, and Single indicates Single Label Word.
TREC
TREC50
GoEmotions
Augmentation Methods
TREC50
Yahoo
64
83.5 (4.1)
76.8 (5.1)
22.1 (2.4)
Random Mask (2019) (ours)
87.9 (5.1)
68.1 (0.9)
hidden
128
85.0 (3.8)
82.5 (6.6)
23.9 (1.I)
Word Delete (2019)
83.4 (4.3)
(1.3)
dimension
85.4 (3.7)
(5.1)
25.4 (2.5)
Word Swap (2019)
(4.2
68.1 (1.0)
(d,,.)
512
84.7 (4.2)
84.0 (9.3)
25.2 (2.1)
Single
Word Delete, Swap (2019)
(5.3)
67.9 (1.3)
BERT-aug (2021)
(5.9)
67.2 (1.4)
Back-trunslation (2020)
85.7 (5.6)
67.5 (0.7)
Table 7: Self-training performances depending the hidden dimension of MAV.
R3F (2021)
87.5 (5.6)
67.5 (1.1)
Cutoff (2020)
(5.9)
(1.1)
Random Augmentation
85.1 (5.9)
67.9 (0.9)
the semantic information of unlabeled samples, is reasonable to construct weakly augmented sam- plcs the same as the original, and let dropout in the model act as minimal data augmentation of hidden representations (Gao al., 2021b).
Auto Augmentation (2022)
85.2
67.7 (1.3)
Table 8: Self-training performances depending
strong augmentation methods. Note that weak augmen- tation fixed with dropout.
On the other hand, strong augmentation plays a significant role in performance, as emphasized by Sohn et al. (2020). Chen et al. (2021) considered only EDA techniques (Crop, Swap, and Deletion) a comparison group for strong augmentation methods. However, given the diversity of augmen- tation techniques in the NLP field, more rigorous comparison experiment is necessary. Thus, we em- ployed the expanded augmentation pool provided by Kim al. (2022). In addition to EDA, contex- tual augmentation and continuous augmentation are added to ensure that a sufficient variety of aug- mentation techniques can be considered. A total of eight augmentation techniques utilized in this study are follows:
BERT-aug (Yi et al., 2021): applying random masking with 15% probability and replacing with the result of pre-trained BERT inference.
Back-translation (Xie et al., 2020): translat- ing each sample to English-
R3F (Aghajanyan et al., 2021): injecting noise sampled from a uniform distribution into word embeddings.
Cutoff (Shen et al., 2020): removing from word embeddings by factor of 0.3.
With these augmentation methods, we designed three comparison experimental settings as follows:
Random Mask (Devlin et al., 2019): ran- domly masking tokens with probability of 15%,
Single Augmentation: applying one augmen- tation technique the augmentation pool,
Random Augmentation:
randomly sampling augmentation techniques per batch (similar to FixMatch's augmentation).
Word Delete (Wei and Zou, 2019): removing each token with probability of 20%,
Auto Augmentation (Kim et al., 2022): up- dating the augmentation policy every iteration to select augmentation techniques that are dif- ficult but not too semantically different from the original.
Word Swap (Wei and Zou, 2019): swapping the position of each token with another token with a probability 20%,
Word Delete/Swap (Wei and Zou, 2019): ran- domly choosing between Word Delete and Swap for each sample,
The results are shown in Table Random Mask achieved the best performance, demonstrating the

---

Dataset
(Class: Single Label word)
AG's News
(World: World), {Sports: Sports), (Business: Business); (Sei/Tech: Teck)
[Abbreviation: Expression], ). (Entity: Entiry). (Description and abstract concept: Description).
TREC
(Human being: Human). (Location: Location). (Numeric value: Number)
(Society&Culture: Sociery), /Schenccs&Maahemmaiiii Science), {Health: Health),
[Educationn&Referennee Education), [Computers&&llleenee Computer). (Sports: Sports).
Yahoo Answers
[Business&FFnance: Business), (Entertainmentt Music: Entertainennt)
[Family&Rclationstin Relationship), (Polities&&Goverrmeett Politics)
[Expression abbreviated:
shortened), (Animal: animal), [Lasting time of something: period), Invention, book and other creative piece: creation), (Discasc and medicine: medical), [Food: food). [Other entity: other), 1, (Sport: sport), |Fquivalent term: equal),
TREC50
(Definition of something: )efinition)) (Description of something: description).
(Manner of an action: manner). [Reason: reason). [Group or organization persons: group). (Individual: individual), {City:: city), ). (Country: country), {Other location: location),
(State: state). [Number of something: count). [Date: date]. (Price: money]
(Optimism: optimism), (Neutral: newiral). [Amusement: amusement), (Curiosity: cariosity).
(Surprisc: surprise), [Confusion: confusion), J. [Nervousness, nervous)] ). (Disgust: disgusr), (Joy:
(Anger: anger). (Gratitue:: gratitude). [Sadness: sadness). (Disappointmmnnt: disappointment).
GoEmotions
[Desire: desire). (Embarrassment: emburrassment). (Remorse: remorse). (Realization: realization),
{ [Excitement: excitement), i, [ AAmmiratio:: admiration). (Disapproval: disapproval), |Caring: caring),
[Fear: fear). (Approval: approval). (Love: love), [Annoyance: annoyance). (Relief: relief)
Table 9: Manually selected single label words for each dataset. The label words the TREC dataset are from LM-BFF (Gao et al., 202 la), and those of the AG's News and Yahoo Answers datasets are from PET (Schick and Schütze, 2021). For the TREC50 and GoEmotions datasets, we manually chose the label word to be a single token that closely represents the class name. Note that each label word is single token including space symbol (e.g.. "GWorld").
suitability of the strong augmentation technique adopted in this study. Prompt-based trf-niran naturally benefits from the random masking tech- nique since it maintains the input form contain- ing the [MASK] token which also used in the pre- training stage.
B.3 Baselines
Single label word For the TREC dataset, we used manual label words provided by LM- BFF (Gao et al., 2021a), and for the AG's News and Yahoo Answers datasets, which were not tested by LM-BFF, we used the manual label words pro- vided by PET (Schick and Schütze, 2021). For the TREC50 and GoEmotions datasets, which were not tested in either of the previous studies, we adopted the class name as a label word. If the class name is not single token, we manually selected a sim- ilar one. All label words used in this study are described in Table 9. Furthermore, we excluded the automatic search method proposed by Gao al. (2021a a) due to increasing training cost proportional to the number of classes. Additionally, we dis- missed the demonstration technique of sampling training instances of all classes individually and concatenating them to the input sequence due to the maximum input length.
Sipeerpparameters for training In this section, we describe the hyperparameter tuning process as- sociated with the training loop. We conducted grid search for the learning rate within the range of {1e-5, 5), the weight of the self-training loss (A1) within the range of (0.5, 1, and the weight of the auxiliary MLM loss (A2) within the range of [0.1, 1]. These ranges were selected based on pilot experiments conducted on the TREC and TREC50 datasets, aiming to cover diverse set of values.
Foral experiments, we trained the models for up to 200 epochs and evaluated the performance every 20 epochs. The best model was selected based on its evaluation performance. To accommodate the GPU memory limitations, gradient accumulation was employed, allowing larger effective batch sizes. The batch size for all experiments was set to 32.
Multi label words AMuLaP (Wang al., 2022a) is more efficient than other automatic verbalizers as it does not require parameter updates to explore label words. Before fine-tuning, MLM inference is

---

TREC
TREC50
GoEmotions
Verhalizer-free
76.0 (6.3)
81.7 (5.5)
23.2 (1.1)
MAV (ours)
85.4 (3.7)
87.9 (5.1)
25.4 (2.5)
Table 10: Self-trainin training performances of verbalizer-fiee method.
performed with a small number of labeled samples to select label words with high prediction proba- bility for each class. The logits of these tokens are then added together and considered the logit value of the class. In this study, we selected the top 16 tokens with high prediction probability as label words for each class, applying deduplication to prevent overlap of label words across classes.
B.4 Training details
We employed the PyTorch² library for training model. The overall organization of the experimen- tal code is derived from the code implemented in previous studies Whole training process of this re- search was conducted using NVIDIA RTX A6000.
Comparison of MAV vs Verbalizer-free Approach
MAV shares similar goals with verbalizer-free methodologies both utilize the [MASK] repre- sentation to construct soft label words. The distinc- tion lies in MAV proceeding with MLM prediction, while most verbalizer- free methodologies rely on spatial distance from the [MASK] representation.
As mentioned in Section 2.2, existing verbalizer- free methodologies are not applicable to self- training algorithms based on pseudo-labeling and consistency regularization, which require explicit output probability distributions. Thus, direct per- formance comparison with the proposed MAV is not feasible. Alternatively, we considered using the [MASK] representation fed into the classification head as a verbalizer-free baseline. The results are shown in Table 10. These findings demonstrate that leveraging all the information in the vocabu- lary logits produced by MLM prediction is more effective for multi-class classification than simply using the [Mask] representation.
'nttps://pytorch.org
"https://githh:" wttp://ninnetooonn/////////////e http://...bbb.cco/////tteees////


Reviews:
[{"review_id": "TaZW2tNo8i", "reviewer": "Reviewer_jSyo", "summary": "", "questions": "a, More details for the \"Multi Label Words Mapping\" step in Fig. 1 is recommended to be provided, as there exit various solutions for this task;\nb, What is the masking strategy difference between the \"Random Masking\" operation for \"Auxiliary MLM loss\" vs. the \"Strong-Aug (Random Masking)\" operation for \"Self-training loss\"?\nc, In lines 229-230, is the low-dimensional vocabulary d_ve learned during the training or a pre-fixed set?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable comments. It's encouraging to hear that you view our work as both promising and feasible for tackling multi-class classification challenges, and that the empirical results have effectively supported our claims. We have carefully considered your main concerns and have addressed them in the following manner:\n\n**Source code for this paper**\n\nWe would like to highlight that we have diligently prepared the source code for reproduction purposes. This comprehensive code, available as Supplementary Materials, is accompanied by detailed documentation(`readme.md`) outlining the experimental procedure and hyper-parameter specifications. We believe that this effort will address any reservations regarding reproducibility you might have.\n\n**More details for Multi Label Words Mapping (*Question A*)**\n\nWe would like to clarify more details for Multi Label Words Mapping. \n\nLabel word mapping is a technique that specifies one or several label words corresponding to each class before fine-tuning. Subsequently, only the predicted values of these label words are extracted from the MLM predictive distribution, constructing the final probability distribution for multi-class classification. If more than one label words are selected from the word dictionary, which has a vocab size of 50265 for RoBERTa-base, it becomes “Multi Label Words Mapping”. The selection of label words can be performed manually or via automatic methods.\n\nFig. 1 (b) visualize the process of Multi Label Words Mapping in multi-class classification task with 6 classes. A set of multiple label words corresponding to each class is defined (e.g., V_Location). The sum of MLM predictions for label words within this set (e.g., *Location, City, Place,* ...) forms the predicted probability for the \"Location\" class. In this paper, we used AMuLaP (Wang et al., 2022a), an automatic label word searching approach, as a baseline for “Multi Label Words Mappping”. AMuLaP involves zero-shot inference before fine-tuning with only a small amount of labeled data (about 16 examples per class) to identify top-k (we set the value 16) words with substantial MLM prediction probabilities. This methodology exhibits strength in binary classification tasks where the distinction between classes is clear. However, as shown in Table 4, under multi-class dataset where the distinction between classes is relatively vague, tokens unrelated to “Location” class such as *Poll* and *Related* are selected as label words, which leads to the poor performance of Multi Label Words Mapping in Table 2. \n\n**\"Auxiliary MLM loss\" vs. \"Strong-Aug (Random Masking)\" (*Question B*)**\n\nWe would like to highlight that the \"Self-training loss\" and the \"Auxiliary MLM loss\" serve different purposes and utilize distinct masking outcomes of unlabeled samples.\n\nThe self-training process endeavors to ensure that the predicted distribution of the strongly augmented samples aligns closely with the pseudo-labels derived from the weakly augmented samples. Hence, in this scenario, random masking exclusively functions as an strong augmentation technique. As depicted in Fig. 2 `Strong-Aug (Random Masking)`, only the masked token within the template, denoted in red, traverse through PLM, MLM head and MAV, generating final predictive distribution used in the computation of the self-training loss.\n\nOn the other hand, for \"Auxiliary MLM loss\", the unlabeled input is independently masked, separate from strong augmentation. The random Masking applied in this context aims to enhance the model's ability to learn diverse contextual relationships by masking tokens in the unlabeled samples. This allows pre-trained language model to preserve the ability to perform MLM during prompt-based fine-tuning process. Therefore, in this scenario, the MLM loss derived from the newly masked token, denoted in red, serves as an auxiliary loss (`Random Masking` at the bottom of Figure 2).\n\n**The dimension of vocab extractor, d_ve (*Question C*)**\n\nDuring the fine-tuning process, all parameters of the two fully connected layers comprising  MAV are learned. The hidden dimension (d_ve) of the first layer, named vocab extractor, is a pre-fixed value (256) determined by hyper-parameter tuning involving four comparison groups ({64, 128, 256, 512}). Details for the tuning can be found in Appendix B.1 and Table 7.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Prompt-based self-training is a promising framework for FSL tasks in NLP. Extending the current works into a multi-class scheme by adopting the automatic verbalizer is an algorithmically feasible solution.\n\nEmpirical results support the author's premise well.", "reasons_to_reject": "Considering the multi-stage training of the MLM head and the multiple loss functions involved in this framework, re-implementation of the work and reproducing the results would be quite challenging. The paper did not provide any source code although it uses public datasets. B.3 might not be as easy as it describes. ", "questions_to_authors": "a, More details for the \"Multi Label Words Mapping\" step in Fig. 1 is recommended to be provided, as there exit various solutions for this task;\nb, What is the masking strategy difference between the \"Random Masking\" operation for \"Auxiliary MLM loss\" vs. the \"Strong-Aug (Random Masking)\" operation for \"Self-training loss\"?\nc, In lines 229-230, is the low-dimensional vocabulary d_ve learned during the training or a pre-fixed set?", "paper_topic_and_main_contributions": "The manuscript proposed a multi-class prompt-based self-training framework with an automatic verbalizer. By utilizing the trained verbalizer, the framework provides a learnable and mapping-free form for self-training. Experiment results demonstrate the effectiveness of the proposed framework by comparison with the latest methods.", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}, {"review_id": "1DYYC0Lvry", "reviewer": "Reviewer_iK6f", "summary": "", "questions": "A. In §4.5, what is the justification for calculating SHAP values only for the correct predictions?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for valuable observations in enhancing the clarity and impact of our paper. We appreciate your interest in our methodology and the robustness of our comparisons, as well as your acknowledgment of the code's availability for result reproduction. We are also pleased that you recognize that our performance numbers are supported by the analyses in §4.3 and §4.5. We address your main concerns as below:\n\n**Why MAV benefits self-training?**\n\nWe value your insightful perspective on the overall narrative of the paper and its focus. We acknowledge the importance of clarifying the motivation behind our emphasis on the interaction of the verbalizer with self-training.\n\nOur paper focuses on the prompt-based self-training task, which involves prompt-based fine-tuning and semi-supervised learning (especially, self-training based on pseudo-labeling and consistency regularization such as FixMatch). In this context, an effective verbalizer must address two aspects: (1) identifying suitable label words for prompt-based fine-tuning and (2) generating explicit probability distributions for self-training. Current manual/automatic verbalizers derive predictive distribution through label word predictions but are inefficient due to the cost of label word selection. The verbalizer-free approach mentioned in §2.1 lacks the ability to derive probability distributions due to distance-based learning.  Our contribution, MAV, addresses these issues by eliminating label word selection overhead in prompt-based fine-tuning and generating explicit probability distributions for self-training. This design bridges the gap between verbalization and self-training, addressing the challenges of both aspects in a unified manner.\n\n**More details for experimental settings and Table2**\n\nAs you clarified, \"Small-supervised\" and \"Full-supervised\" in Table 2 correspond to the models trained with the supervised loss outlined in §3.2, while \"Semi-supervised\" represents the model trained with both self-training loss and auxiliary MLM loss in addition to supervised loss. This distinction will be explicitly added to the text and table captions for clarity.\n\nFurthermore, within the context of Table 2, the amalgamation of \"Small,\" \"Semi,\" \"Full,\" and the \"Benefit ratio\" metrics serves a crucial purpose for grasping the efficacy of self-training. \"Small-supervised\" showcases how \"Semi-supervised\" benefits from incorporating unlabeled data with limited labeled data, while \"Full-supervised\" represents the upper performance limit in the semi-supervised context. Moreover, the \"benefit ratio\" metric provides a quantitative means to comprehend the interplay between these configurations and the effect of self-training.\n\nRecognizing the need for enhanced clarity, we will consider reorganizing Table 2, separating \"Small,\" \"Semi,\" and \"Full\" into distinct subtables for intuitive comparison.\n\n**Effect of freezing the MLM head**\n\nWe appreciate your comment regarding the comparison between freezing the MLM head and freezing the entire encoder in §4.4. We understand your point about the difference in the scale of trainable parameters between these two scenarios, which indeed presents a nuanced aspect to consider.\n\nYour observation about the conclusion drawn from Table 3 is also well-taken. Our objective in freezing the MLM head is to generate informative predictions that are not biased by the small amount of labeled data during the fine-tuning process. Simultaneously, we harness the pre-trained knowledge embedded within the model. Nonetheless, we acknowledge that while the results show that freezing the MLM head doesn't negatively impact performance, the direct inference that it can produce sufficiently informative MLM predictions for downstream data requires further substantiation. In our revised manuscript, we will aim to address the comparison more judiciously by emphasizing the distinction in parameter scales between the scenarios.\n\n**In §4.5, what is the justification for calculating SHAP values only for the correct predictions? (*Question A*)**\n\nThe reason for this approach is rooted in the nature and intent of §4.5, which focuses on post-hoc analysis rather than performance evaluation. The primary aim is to gain insights into the mechanisms by which MAV identifies relevant vocabulary features for classification, rather than assessing overall model performance.\n\nGiven this context, we deemed it appropriate to calculate SHAP values solely using data with correct predictions. Excluding incorrect answer data minimizes the potential noise introduced by instances where the model's prediction might not align with the actual class associations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Overall, the methodology and the comparisons to different baselines look sound, and code for reproducing the results is provided.\n- The authors demonstrate strong few-shot performance for the proposed approach - that does not require any manual label names - even outperforming baselines that do require a manual label. The performance numbers are also backed by analyses that show the approach induces better separation of mask representations (§4.3) and results in logical verbalizer behavior (§4.5).\n- The approach of adding layers on top of the MLM head is interesting. In a sense this is a combination between classic fine-tuning - where we simply train a new classification head that converts model outputs to class labels, and prompt-based fine-tuning - where we rely on the knowledge of the MLM head to map to a set of class label names. So here we still rely on the MLM knowledge to some extent, but nevertheless need to learn the task-specific mapping on top of that.\n", "reasons_to_reject": "- In my view there is a certain lack of focus in the overall narrative of the paper, in terms of more precisely describing what is the problem the paper aims to solve and what is the unique contribution of this work. Specifically, the problem of implementing a good verbalizer for (multi-class) classification is a relatively general one, whereas the question of the interaction of the verbalizer with self-training is more specific. There is no problem with focusing more on the latter, except I did not find in the paper either a very clear motivation for this — i.e., why would self-training specifically suffer more from a suboptimal verbalizer — or an interesting analysis of this point — i.e, showing how something about the way MAV works explains why it benefits self-training specifically.\n- I found the description of the experimental setting and baselines a bit difficult to follow, in that it is not sufficiently clear to the reader what is kept constant and what differs between the settings reported in Table 2. For instance, if my understanding is correct, all the rows marked with \"Semi-supervised\" use the loss formulations as described in §3.2; this is not sufficiently clear from the text.\n On a related note, I think the structure of results in Table 2 can be improved, as it makes it hard for the reader to see the comparison between different approaches over the same setting (as these rows are not adjacent to each other). For example, what is arguably the core result in this paper - that MAV outperforms the closest prior work of SFLM, gets a bit lost in the table. Put another way, if my problem is text classification and what I have are few-shot labeled examples, what I would want to know is which method works best in this scenario; if I also have unlabeled data and can do ST, I want to know what works best for that; and if I have a lot of labeled examples, then again I would want to know which approach to choose, and the results can be presented in a way that more clearly conveys this information. In contrast, knowing what is the \"benefit ratio\" for each method is less useful, assuming I am interested in dealing with a specific scenario (e.g. few-shot) and want to reach a better absolute performance on a downstream task.\n- Comparing freezing the MLM head to freezing the entire encoder (§4.4) is a bit odd, as it is not the same scale of trainable parameters. In any case, while it is clear that freezing the MLM head does not hurt performance, concluding from this that it is \"able to produce MLM predictions that are sufficiently informative about the downstream data on its own\" (l. 475) is not really substantiated by Table 3.\n", "questions_to_authors": "A. In §4.5, what is the justification for calculating SHAP values only for the correct predictions?", "paper_topic_and_main_contributions": "This paper deals with the problem of few-shot multi class classification, specifically in the context of performing the classification using the masked language modeling capabilities of an encoder language model. The authors propose a new approach for a learnable verbalizer that converts the masked token predictions of the model to class predictions. The approach consists of adding two layers on top of the MLM head for learning the classification task with few-shot labeled data, while keeping the pre-trained MLM head frozen. The authors combine their verbalizer approach with existing approaches for self-training in the context of prompt-based fine-tuning, and compare this with baseline verbalizer methods over 5 multi class datasets. The main contribution is in proposing the new approach for a learnable verbalizer for few-shot classification, which importantly does not require manually specifying class label names; the authors show that their approach outperforms existing verbalizer approaches for few-shot prompt-based fine-tuning, especially where self-training is used.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "l. 130: answer to a critical research -> answer a critical research\nl. 142: of manual approach -> of the manual approach \nl. 237: MAV autonomously identify -> MAV autonomously identifies"}, "extra_text": {}}, {"review_id": "q0P9ZQLOQZ", "reviewer": "Reviewer_jFV1", "summary": "", "questions": "A. Why is there no evaluation on larger LMs such as RoBERTa-large (355M parameters) as the previous work (Gao et al. 2021a) did? Fine-tuning becomes much more expensive as the size of LMs grows and few-shot fine-tuning plays a critical role especially when fine-tuning larger LMs. Since the paper experiments in a few-shot setting, an evaluation on larger LMs is recommended.\n\nB. How sensitive is the proposed method to the value of K, the number of examples per class? It would be interesting to compare it with other baselines.\n\nC. Why is there no evaluation using the entire dataset? It is assumed that the proposed method can be fine-tuned on the entire dataset without filtering. This experiment is needed to evaluate the applicability of the proposed method to any multi-class datasets.\n\nD. How is the test set for each dataset created? Given that the datasets are pre-processed based on the number of classes, as shown in Table 1, it is supposed that the test set is also pre-processed. Furthermore, while the train and dev sets are class-balanced, is the test set the same?\n\nE. Why is the fine-tuning process so long? In general, the number of epochs is set to a small value, such as 3, 5, or 10, in fine-tuning (Gao et al. 2021a; Liu et al. 2019). However, the paper uses 200 for the number of epochs (Line 977), which seems to be a bit larger value.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We extend our gratitude for taking the time to provide us with invaluable comments and inquiries regarding our paper. Your acknowledgment of our method as being well-motivated, and your recognition of the paper's clear structure and comprehensive descriptions, are appreciated. The detailed responses to your main concerns and questions are as below:\n\n**Why is there no evaluation on larger LMs such as RoBERTa-large (355M parameters) as the previous work (Gao et al. 2021a) did? (*Question A*)**\n\nWe would like to clarify our rationale for the model selection in our study. Prior investigations in the realm of prompt-based fine-tuning have employed a range of pre-training language models. Notably, these include Gao et al. (2021a) and Hu et al. (2022) with RoBERTa-large, Chen et al. (2021) with RoBERTa-base, and Zhao and Yao (2022) with DistilRoBERTa. In our study, we deliberately opted for RoBERTa-base to facilitate optimal comparisons with the most closely related prompt-based self-training literature, the SFLM approach of  Chen et al. (2021). While earlier research in prompt-based fine-tuning with only limited labeled data might have sought to harness the advantages of larger models to enhance few-shot fine-tuning efficacy, our emphasis was on the self-training process with additional unlabeled data rather than the scale of the model.\n\n**How sensitive is the proposed method to the value of K, the number of examples per class? (*Question B*)**\n\nDrawing on insights from prior prompt-based fine-tuning studies (Gao et al., 2021a; Chen et al., 2021), it's evident that employing a smaller K (i.e., fewer labeled samples) can yield significant performance advantages over classical fine-tuning. While these studies have observed performance variations with varying K values, their experimental setup is conducive to constructing datasets for comparative analysis due to their focus on binary or 3-class classification scenarios.\n\nHowever, our study faces a unique challenge in this context. The limited amount of available data impedes our ability to conduct experiments with varying K values. A crucial constraint emerges from the necessity to omit classes with insufficient data to achieve a balanced setting, thereby hindering the establishment of a fair comparison framework.\n\n**Why is there no evaluation using the entire dataset? (*Question C*)**\n\nAs elaborated in Section 4.1.1, the decision to exclude certain classes from datasets like TREC50 and GoEmotions was rooted in the necessity to maintain a balanced self-training framework, given the minimum data requirement of 80 examples per class. This choice aimed to ensure the integrity of training, as extremely small sample sizes for certain classes could hinder robust model learning and data consistency. Some of the eliminated classes consist of only a handful of samples, as low as 4 or 6. \n\nAddressing such imbalanced setting lies beyond our current scope. We acknowledge it in the Limitations section and propose it as potential future work. To openly handle this concern, we plan to add appendix with a table detailing the classes that were excluded from each dataset, enhancing clarity for readers.\n\n**How is the test set for each dataset created? While the train and dev sets are class-balanced, is the test set the same? (*Question D*)**\n\nSince it is not possible to make predictions for classes that are not included in the training, classes that are removed from the train and dev datasets are correspondingly excluded from the test dataset.\n\nRegarding class balance, the characteristics of test sets differ across datasets. In the case of Yahoo Answers and AGNews, the test sets are intentionally designed to maintain a balanced distribution of classes. Conversely, TREC, TREC50, and GoEmotions exhibit class-imbalanced test sets, reflective of their original nature. We will supplement the information about the test datasets in order to enhance the overall clarity of our work.\n\n**Why is the fine-tuning process so long (200 epochs)? (*Question E*)**\n\nWe selected the number of training epochs based on careful consideration of existing literature. Gao et al. (2021a) and Chen et al. (2021) employ a maximum step of 1000, with validation conducted every 100 steps. Taking into account details, where the number of labeled samples per class (k) is 16 and the maximum batch size used for hyper-parameter tuning is 8, the training epoch range is estimated to be around 160 to 250 epochs for binary or 3-class classification.\n\nGiven the increase in the number of classes in our dataset compared to previous studies, we opted not to simply replicate the same max step value. Instead, we set 200 epochs as the maximum number of training iterations. We also set the validation interval to 20 epochs to ensure equal intervals.\n\nIt's important to highlight that prompt-based fine-tuning diverges from regular fine-tuning in that it doesn't operate with the full dataset, contributing to the relatively extended process duration.\n\n**More details for Section 4.4 and Table 3**\n\nThe experiment in Section 4.4 was designed to compare and analyze the impact of freezing the MLM head, a key feature of how MAV works. This freezing mechanism aims to capitalize on the pre-trained vocabulary representation inherent in the decoder of the MLM head. The three settings shown in Table 3 are: updating all parameters, freezing only the MLM head, and freezing the RoBERTa encoder. Basically, we compared the performance of prompt-based self-training based on the MAV structure with different parameter update scenarios while following the overall self-training framework in Figure 2. \n\nAs evidenced in Table 3, the results of freezing the MLM head's parameters closely align with those achieved through full fine-tuning. At the same time, it significantly outperforms the scenario where the RoBERTa encoder is frozen. This suggests that the MLM head is sufficiently adept at generating MLM predictions that carry valuable information about the downstream data even without further tuning.\n\nIn response to your feedback, we will enhance the clarity of Section 4.4 by elaborating on the experimental setting surrounding Table 3.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed method is well-motivated and addresses issues of manual verbalizer creation that requires domain knowledge and an increasing cost for automatically searching label words correlating with the number of target classes.\n2. The paper is well-structured and easy to follow. Sufficient descriptions of the experimental setup are provided, which ensures reproducibility.", "reasons_to_reject": "1. The paper conducts the experiments only with a single pre-trained language model, RoBERTa-base, which is also the smallest one (125M parameters) in size among RoBERTa variants, and a single few-shot setting (K=16), both of which narrow the generalizability of the proposed method.\n2. The paper argues that the proposed method can be applied to any multi-class datasets, irrespective of the number of classes (Lines 109-110), while the experiments are conducted on filtered datasets, as shown in Table 1, where the minor classes are removed (e.g., the number of classes decreased from 50 to 22 in Trec50) to meet the requirement regarding the number of examples per class to fine-tune the proposed method. This may contradict the argument and limit the applicability of the proposed method to any multi-class datasets.", "questions_to_authors": "A. Why is there no evaluation on larger LMs such as RoBERTa-large (355M parameters) as the previous work (Gao et al. 2021a) did? Fine-tuning becomes much more expensive as the size of LMs grows and few-shot fine-tuning plays a critical role especially when fine-tuning larger LMs. Since the paper experiments in a few-shot setting, an evaluation on larger LMs is recommended.\n\nB. How sensitive is the proposed method to the value of K, the number of examples per class? It would be interesting to compare it with other baselines.\n\nC. Why is there no evaluation using the entire dataset? It is assumed that the proposed method can be fine-tuned on the entire dataset without filtering. This experiment is needed to evaluate the applicability of the proposed method to any multi-class datasets.\n\nD. How is the test set for each dataset created? Given that the datasets are pre-processed based on the number of classes, as shown in Table 1, it is supposed that the test set is also pre-processed. Furthermore, while the train and dev sets are class-balanced, is the test set the same?\n\nE. Why is the fine-tuning process so long? In general, the number of epochs is set to a small value, such as 3, 5, or 10, in fine-tuning (Gao et al. 2021a; Liu et al. 2019). However, the paper uses 200 for the number of epochs (Line 977), which seems to be a bit larger value.", "paper_topic_and_main_contributions": "This paper focuses on a prompt-based fine-tuning on multi-class text classification, specifically where the number of classes is greater than or equal to 4. Given that creating a manual verbalizer is expensive and time-consuming and that automatically searching for optimal label words is computationally costly, the paper proposes a method called mapping-free automatic verbalizer (MAV) on top of a previous prompt-based self-training method, SFLM (Chen et al., 2021). The proposed method circumvents the aforementioned issues by appending two learnable fully-connected layers to the MLM head of RoBERTa. Experimental results demonstrate the effectiveness of the proposed method in a few-shot setting on five datasets, where the number of training samples K is set to 16, and the number of classes ranges from 4 to 26. The paper also provides a number of analyses to support the superiority of the proposed method over the comparison methods.\n\nUpdated: I have read the author rebuttal and keep the scores unchanged. While the rebuttal makes sense, current experimental results with only RoBERTa-base and K=16 raise the question about the generalizability of the proposed method with other language models with different sizes as well as different values of K.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Section 4.4 and Table 3 are somewhat difficult to follow. The experimental setting on Table 3 should be described. Does this experiment study the effect when different parts of the model parameters are frozen?"}}]

Author rebuttal:

Meta-review: This paper presents a method for multi-class few-shot and semi-supervised classification with masked language models. Previous approaches use manually defined verbalizers that map from class labels to tokens in the MLM’s vocabulary, but this does not scale well to datasets with more classes. The method instead adds a classification head on top of the distribution over tokens outputted by the MLM head. Experiments are on text classification tasks and often combine the method with self-training on unlabeled examples. Reviewers found the method appropriate and appreciated the strong empirical results and additional analysis. As pointed out by reviewer jFV1, the experiments are not the most thorough: only covering one pretrained model and a fixed number of labels per class. It also isn’t justified why going through the vocabulary is necessary at all as opposed to applying the classification head to the [MASK] hidden state directly. Additionally, the experiments filtered out classes from the datasets with very few examples per class, which I find strange because this is precisely the setting where we want good few-shot learning. Lastly, as reviewer iK6f suggests, the paper could be clearer about the problem being solved: while the paper largely discusses few-shot learning, the results for the pure few-shot setting (with no unlabeled data) are often worse than the single word label baseline. Instead, the method really shines in the semi-supervised setting, but there isn’t much analysis on why it benefits more from self-training than other approaches. To summarize, the method is well-motivated and produces some strong results, but the paper could be clearer about when and why the method works.
Decision: Accept-Findings