Prototype-Guided Pseudo Labeling for Semi-Supervised Text Classification
Year: 2023

Abstract:
The semi-supervised text classification (SSTC) task aims at training text classification models with a few labeled data and massive unlabeled data. Recent works achieve this task by pseudo-labeling methods that assign pseudolabels to unlabeled data as additional supervision. However, these models may suffer from incorrect pseudo-labels caused by underfitting of decision boundaries and generating biased pseudo-labels on imbalanced data. We propose a prototype-guided semi-supervised model to address the above problems, which integrates a prototype-anchored contrasting strategy and a prototype-guided pseudo-labeling strategy. Particularly, the prototype-anchored constrasting constructs prototypes to cluster text representations with the same class, forcing them to be high-density distributed, thus alleviating the underfitting of decision boundaries. And the prototype-guided pseudo-labeling selects reliable pseudo-labeled data around prototypes based on data distribution, thus alleviating the bias from imbalanced data. Empirical results on 4 commonly-used datasets demonstrate that our model is effective and outperforms state-ofthe-art methods.

heading: 1 Introduction
text: Traditional text classification has achieved profound success by leveraging numerous labeled data. However, acquiring large-scale labeled data is expensive in many real-world scenarios, making training effective classifiers using such approaches difficult. Therefore, semi-supervised text classification (SSTC), aiming at text classification requiring a few labeled data and massive unlabeled data, has become a new appealing technique.
As a promising semi-supervised paradigm, the pseudo-labeling method (Mukherjee and Awadallah, 2020; Xie et al., 2020; che; Lee et al., 2021; Li
∗Corresponding author: zhangrc@act.buaa.edu.cn
et al., 2022), which assigns pseudo-labels to unlabeled data instances as additional supervision, has drawn attention. They follow the well-known lowdensity separation assumption (Chapelle and Zien, 2005) to generate pseudo-labels for the unlabeled data scattered in the high-density feature space, helping to identify the boundaries between classes. Trained with the mixture of labeled and pseudolabeled data, the model perceives more abundant supervision signals and reduces empirical errors. Existing models attempt to assign soft pseudolabels for unlabeled texts (Xie et al., 2020) or keep pseudo-labels with lower uncertainty (Mukherjee and Awadallah, 2020) to alleviate the influence brought by the incorrect pseudo-labeled data.
In practice, the underfitting of decision boundaries is usually detrimental to the model learning (shown in Figure 1(a)). More careful considerations are expected in properly utilizing these pseudo-labeled data near decision boundaries. In addition, the pseudo-labeling process may meet with bias from imbalanced data and will be biased toward dominant class data (shown in Figure 1(b)). This problem may further lead the dominant classes to dominate the follow-up training and assign more incorrect labels to the unlabeled data.
Believing that addressing both the underfitting of decision boundaries and the bias from the im-
16369
balanced data is crucial, we propose a prototypeguided pseudo labeling (PGPL) model to tackle the two problems simultaneously. PGPL improves the SSTC performance with a Prototype-Anchored Contrasting (PAC) module and a Prototype-Guided Pseudo Labeling (PGP) module. Specifically, in the PAC module, we leverage the prototypical learning (Bien and Tibshirani, 2011) to construct prototypes with representations of labeled texts and classify unseen texts according to their distance from the prototype of each class. In this way, we force the text representations to locate in high-density areas, making the unreliable pseudo-labeled data near the boundary away from decision boundaries to address the problem, as demonstrated in Figure 1(a). Furthermore, to alleviate the bias from the imbalanced data, our PGP module chooses reliable pseudo-labeled data by reducing the probability of assigning labels of dominant class for unlabeled data, and thus prevents the model from biasing on dominant classes, as demonstrated in Figure 1(b).
Finally, we conduct experiments on four commonly-used datasets, and the results demonstrate that our proposed PGPL model outperforms the state-of-the-art performance in various semisupervised settings. The experimental analysis also suggests that our PGPL model successfully separates the different classes and assigns reliable pseudo-labels to the unlabeled instances. These results confirm the effectiveness of PGPL.
To summarize, our main contributions are in the following four folds:
• We propose a prototype-guided pseudolabeling model for the semi-supervised text classification task, which simultaneously addresses the underfitting of decision boundaries and the bias from imbalanced data.
• We present a prototype-anchored contrasting module to generate high-density distributed representations that reduces the unreliable pseudo-labeled data chosen for training the semi-supervised models.
• We leverage a prototype-guided labeling module to choose data around prototypes that alleviates the bias from imbalanced data.
• Empirical studies indicate that our proposed model outperforms previous state-of-the-art methods on four commonly-used benchmarks and comprehensive analysis confirm the effectiveness of PGPL.

heading: 2 Related Works

heading: 2.1 Semi-supervised Text Classification
text: Existing methods for SSTC task can be divided into three groups: Generative methods, Graph-based methods and Pseudo-Labeling methods.
Generative methods (Croce et al., 2019, 2020; Gururangan et al., 2019) attempt to learn a generative model that establishes a mapping function based on the distribution of the labeled/unlabeled data to the classes. However, they usually require the labeled and unlabeled data strictly obey the same data distribution, which can hardly be satisfied in practical scenarios.
Graph-based methods (Linmei et al., 2019; Li et al., 2021; Cui et al., 2022) attempt to construct an graph structure underlying the data, where each node represents a labeled/unlabeled instance and each edge represents a pairwise relation in certain similarity measurement. However, they construct such graphs upon all the data, inevitably requiring huge computational resources, which can be impractical on large-scale datasets.
Pseudo-Labeling methods learn the model on the mixture of labeled data and unlabeled data (annotated with pseudo-labels), thus accessing to additional beneficial supervision. There exist two major practices: One line of the methods (Wang et al., 2021; Mukherjee and Awadallah, 2020; Lee et al., 2021; Tsai et al., 2022) is based on self-training strategy , which first trains a base model on labeled data, and then retrains the model on the unlabeled data annotated with pseudo-labels of high confidence. Another line of the methods (Najafi et al., 2019; Xie et al., 2020; Sohn et al., 2020) incorporates the idea of consistency learning, which adopts the unlabeled data to enhance model robustness with data perturbation, such as adversarial attack (Najafi et al., 2019) and data augmentation (Xie et al., 2020; Sohn et al., 2020; Chen et al., 2020). Although these methods provide potentiality and flexibility in practices, they still suffer from the underfitting near decision boundaries problem, and easily encounter learning bias from imbalanced data. In this paper, we develop the previous pseudo-labeling studies with prototype learning, which enhances low-density separation by exploiting prototypes to cluster text representations, and alleviates imbalanced class bias with prototype-guided pseudo labeling.

heading: 2.2 Prototypical Learning
text: Prototypical learning (Bien and Tibshirani, 2011) aims to learn representative prototypes to summarize the instances in the same class, and is been widely used for unsupervised (Li et al., 2020; Pan et al., 2019) and semi-supervised learning (Snell et al., 2017; Arik and Pfister, 2020; Gu, 2020; Ren et al., 2018). In unsupervised scenarios, the most representative method is PCL (Li et al., 2020), which introduces a prototype as the center of mass of a cluster formed by similar samples and assigns each sample to multiple prototypes of different granularity. The goal of training is to bring each image embedding closer to its associated prototype. In semi-supervised scenarios, the prototypes can first be estimated with the labeled instances. Then, the pseudo-labels or classes of unlabeled data are obtained by calculating distances to the prototypes. Among these works, Gu (2020) attempts prototypebased pseudo-labeling for image classification but neglects the underfitting near the decision boundary and bias from the imbalanced data. We utilize prototypes to conduct low-density separation and data selection to alleviate both issues.

heading: 3 Problem Formulation
text: In semi-supervised text classification, we are given a small set of labeled text data and a large set of un-
labeled text data. Formally, let C be the set of all labels of interest. Dl = { (xl1, yl1), · · · , (xlm, ylm) } is the set of pre-annotated m labeled text instances, where each xli and y l i ∈ C represent a text instance and its corresponding label. Du = { xu1 , · · · , xun } is the set of n unlabeled text data, where xui ∈ Du denotes an unlabeled text instance. Semisupervised text classification hopes to leverage the limited annotated data and expand its knowledge to the unlabeled data for pseudo-labeling and include them as additional training data.

heading: 4 Methodology
text: We propose a semi-supervised text classification model PGPL to tackle problems of underfitting near the decision boundary and bias from the imbalanced data. In this section, we first illustrate the overall model structure and introduce the fundamental SSTC structure with supervised and unsupervised losses, then describe the prototypeanchored contrasting module and finally explain the prototype-guided pseudo labeling module.

heading: 4.1 Model Structure
text: The model structure of PGPL is illustrated in Figure 2. The labeled and unlabeled text instances are first input to the Encoder. Then, the labeled text embeddings are used to generate pseudo la-
bels in Prototype Extraction module and encourage each instance closer to its categorical prototype and away from the other categorical prototypes via prototype-anchored contrasting. Finally, we perform prototype-guided pseudo labeling on the unlabeled data. The pseudo-labeled data is used to augment the training set and retrain the model.

heading: 4.2 Fundamental SSTC Framework

heading: 4.2.1 Backbone Text Classifier
text: To utilize the power of pre-trained language models in SSTC, we follow the previous practice (Chen et al., 2020) that uses BERT (Devlin et al., 2019) as the text encoder. Specifically, let x represent either a labeled instance xl ∈ Dl or an unlabeled instance xu ∈ Du. The function f(x; θ) denotes the BERT encoding process with mean pooling operation over the token representations. Afterward, a two-layer MLP with tanh(·) activation is adopted to derive the relevance score of the input text x corresponding to each class:
g(x, ϕy, θ) = MLP(f(x; θ);ϕc), (1)
where ϕc is the MLP parameter corresponding to class c ∈ C. Note that this backbone text classifier can be trained by two objectives: 1) a supervised loss function Ls (in Eq. (3)) over the labeled instances; 2) an unsupervised loss function Lu (in Eq. (5)) over the unlabeled instances.

heading: 4.2.2 Annealing Supervised Loss
text: Since the number of labeled instances is usually limited and costly in SSTC, we adopt the annealing supervised loss to fully exploit the labeled data. There is an existing work that introduces the training signal annealing (TSA) strategy (Xie et al., 2020), which releases labeled instances gradually during the training phase instead of observing them at once, to prevent the model from early over-fitting on the labeled data. Intuitively, in the initial training phase, we expect to learn about the data without overconfidence, in order not for the model to overvalue the incidental features of the data. Therefore, we employ a threshold ηt, and at training iteration t, we select the instances with the highest score of class confidence lower than the threshold ηt for loss calculation. The threshold ηt is derived as:
ηt = t
T (1− τ) + τ, (2)
where T is the total iterations of the training process, and τ is the initial threshold that is set to a
scalar larger than 1|Y| to all the classes. This strategy encourages the model to observe and learn the data with relatively lower confidence, and gradually adapt to perceive all training instances.
Based on the threshold, we define the supervised loss at t iteration with cross-entropy:
Ls=− 1
m
m∑
i=1
I(pyli <ηt) log exp(g(xli, ϕyli , θ)) ∑ c∈C exp(g(xli, ϕc, θ)) ,
(3) where pyli is the predicted probability of y l i corresponding to instance xli, and I(·) is a binary indicator which equals 1 if pyli < ηt or 0 otherwise.

heading: 4.2.3 Selective Unsupervised Loss
text: To leverage the unlabeled text data, the selective unsupervised loss module assigns pseudo-labels for each unlabeled text instance xui ∈ Du to augment the data. Following recent works (Xie et al., 2020; Mukherjee and Awadallah, 2020; Sohn et al., 2020) in semi-supervised learning, the original unlabeled data is augmented with the pseudo-labeled data and used to optimize the text classifier.
Specifically, we incorporate the augmented data x̃ui of x u i with its pseudo label ŷ u i into the new training dataset to train the classifier with crossentropy loss. Formally, for a unlabeled instance xui , its pseudo label is determined by:
ŷui = argmax c∈C exp(g(xui , ϕc, θ)), (4)
Among the unlabeled instances with pseudolabels, we further select high-quality pseudolabeled instances that are used in the next training phase by the following formula:
Lu = − 1
n
n∑
i=1
I(xui ) log exp(g(x̃ui , ϕyui , θ))∑
c∈C exp(g(x̃ui , ϕc, θ))
,
(5) where the binary indicator I(·) ∈ {0, 1} is:
I(xui ) = I(ŷui = c) ∧ I(d(f(xuj ), zc) ≤ dc), (6)
where ∧ represents the logic operation AND. d(·, ·) denotes the distance function that measures the distance between the text representation f(xuj ) and prototype zc of class c. dc serves as the threshold to determine the value in the binary indicator. The selected instances will be used in the next training phase as additional training data.

heading: 4.3 Prototype-Anchored Contrasting Module
text: To alleviate the underfitting near decision boundaries problem, inspired by the low-density separation hypothesis, the prototype-anchored cluster contrasting module (PAC) forces each text representation to be closer to its categorical prototype and far away from unrelated prototypes. As such, the number of text instances lie near the decision boundaries can be reduced.

heading: 4.3.1 Categorical Prototypes Extraction
text: The prototype of a class is defined by the mean vector of text embeddings within the class. Concretely, we first obtain the text embedding f(xli) for each xli ∈ Dl, and generate prototypes according to their classes. For each c ∈ C, the prototype zc is computed by:
zc = 1
nc
∑
yli=c
f(xli), (7)
nc = ∑
yli∈Dl
I(yli = c), (8)
where nc is the instance number of class c.

heading: 4.3.2 Prototype-Anchored Contrasting
text: After we obtain the prototype of each class, for a subset of labeled instances xli ∈ Dl and unlabeled instances xuj ∈ Du, we expect the embedding f(xli) (resp. f(xuj )) to be closer to its prototype zyli (resp. zŷui ) and away from the others. For the unlabeled text instances, we migrate them within the representation space to make each cluster more distinct based on the prediction confidence. More specifically, a confidence score for each instance is calculated and is used as a weight value for determining how much it should be migrated. In other words, as the instances obtained low confidence are more likely to be assigned with incorrect pseudo-labels, how far the model allows the instances to migrate is proportional to confidence score.
Thus, the loss function of this module can be formulated as:
Lp=− 1 m
m∑
i
∑
c I(yli = c) log exp (−d(f(x̃li), zc))∑ k∈C exp(−d(f(x̃li), zk))
− λ n
n∑
j
∑
c
pŷuj I(ŷ u j = c) log exp (−d(f(x̃uj ), zc))∑ k∈C exp(−d(f(x̃uj ), zk)) ,
(9)
where m and n are the instance number of labeled and unlabeled data, respectively. The function d(·, ·) is a distance metric function and λ is a weight factor. We use Euclidean distance for the distance metric between each instance and the prototype embedding, and use softmax normalization for the instances in the representation space. Here we also consider the pseudo-label confidence pŷui (see Eq. (4)) to balance the training.

heading: 4.4 Prototype-guided Pseudo Labeling Module
text: As discussed in the introduction, the unlabeled instances are easily misguided by the dominant classes, inevitably leading to assigning incorrect pseudo-labels. To alleviate the bias from imbalanced data, we propose a prototype-guided pseudo labeling module (PGP). The idea is to select different k-nearest pseudo-labeled instances corresponding to the center (i.e., the prototype) for each class according to a short training history.
To balance the training process, we expect not to select (or select fewer) pseudo-labeled instances for a class that has been over-trained in previous iterations. Thus, we record a statistic value µct during training, which returns the number of selected pseudo-labeled texts for class c at the tth iteration. For simplicity, we use µc<t to denote the summation of µct′ (with t
′ < t) from a training history of previous iterations. Then, at each iteration t, we can find the least trained class and compute its total number of selected pseudo-labeled texts during the
training history by:
µct = ∑
x̃uj ∈Bu
I(ŷj = c), (10)
γt = argmin c∈C
µc<t. (11)
We hope this least trained class utilizes more pseudo-labeled instances in the next training iteration than other classes to keep balanced training. To this end, we determine the number of nearest text instances kc for class c ∈ C according to:
kc =    µct if µ c <t − γt = 0
µct − (µc<t − γt) if 0 ≤ µc<t − γt < µct 0 if µc<t − γt ≥ µct ,
(12) Here we encourage to select more pseudo-labeled instances for the less-trained classes, and fewer pseudo-labeled instances for the over-trained classes, thus alleviating the imbalanced training problem. Based on kc, we select the top-kc pseudolabeled instances near the categorical prototype of class c by Euclidean distance d(·, ·) as follows:
dc = TopK(d(f(x̃uj ), zc), kc), (13)
where dc is the distance threshold used in Eq. (6) to select pseudo-labeled texts. By selecting knearest pseudo-labeled text instances around the corresponding categorical prototypes, the model obtains more reliable training data for SSTC.
The overall objective L is combined with three losses, including the cross-entropy losses on the labeled and unlabeled data, and the prototypeanchored contrastive loss:
L = Ls + β1Lu + β2Lp, (14)
where β1 and β2 are harmonic factors to balance the weight of the different losses and β1 is usually fixed to 1.

heading: 5 Experimental Setup

heading: 5.1 Dataset and Pre-processing
text: We conduct intensive experiments on four widely used text classification benchmark datasets: IMDB (Maas et al., 2011), AG News (Zhang et al., 2015), Yahoo! Answers (Chang et al., 2008), and DBpedia (Lehmann et al., 2015). We follow the original setting of test set splitting and randomly sample from the training set to form the labeled
and unlabeled sets used for the training. The experimental results are averaged after five runs. The statistics and splitting information of the dataset are shown in Table 1.

heading: 5.2 Baselines
text: To verify the effectiveness of our method, we compare our method to the several baselines:
BERT (Devlin et al., 2019) adopts a pre-trained model for text classification, trained with only labeled data. Delta-training (Jo and Cinarel, 2019) distinguishes the instances with incorrect prediction and correct prediction by two separate classifiers. VAMPIRE (Gururangan et al., 2019) pretrains a variational autoencoder, and uses its internal states as features in a downstream classifier. UDA (Xie et al., 2020) makes soft predictions with original data to train the augmented data. MixText (Chen et al., 2020) makes predictions by mixing the original data with the augmented data. UST (Mukherjee and Awadallah, 2020) makes predictions by selecting pseudo labels with the same prediction multiple times. SALNet (Lee et al., 2021) makes predictions with the help of categorical dictionary. FLiText (Liu et al., 2021) adopts A lighter framework with convolution networks, predicted pseudolabels through knowledge distillation. CEST (Tsai et al., 2022) proposes a certaintydriven sample selection method and a contrastenhanced similarity graph in self-training. SAT (Chen et al., 2022) trains a meta-learner to predict the labels with weakly-augmented data.
For fairness, we implement BERT, UDA, MixText with the code provided by Chen et al. (2020) in our experiments. For other baselines, we use the reported results provided from the original research paper to avoid re-implementation bias.

heading: 5.3 Implementation
text: We used PyTorch1 for implementation. For all compared models, the maximum sentence length is set to 256. For sentences that exceed the limit, we keep the first 256 tags. All hyper-paramters are selected by grid search on the validation set. We choose 1e-5 as the learning rate for the BERT encoder and 1e-3 for the MLP. For the unlabeled data, we choose German as the intermediate language
1https://pytorch.org/
for back translation augmentation using FairSeq22, with a random sampling temperature of 0.9. We use AdamW (Loshchilov and Hutter, 2018) to optimize model parameters. The train epoch number is set to 20. In the training process, we choose 16 unlabeled samples for each batch. And for the labeled data, the number of labeled data is the number of classes of each dataset in each batch. The factor λ is set to 0.2 in IMDB and AG News and 0.5 in Yahoo!Answer and DBpedia. The history t iteration is set to 5. The harmonic factor β1 and β2 are set to 1 and 0.5, respectively.

heading: 5.4 Overall Results
text: We first compare our model with the BERT, UDA and Mixtext. We run these models with the same setting of our model. The training set consists of 5000 unlabeled data and respectively 10, 30, and 200 labeled data for each class.
Table 2 shows the results on 5000 unlabeled data and different amount of labeled data. The results show that our method is superior to all baselines. Especially when there are only 10 labeled instances for each class, our model outperforms by 6.8% for AG News, 18.3% for IMDB, and 7.3% for Yahoo Answers, respectively. On DBpedia, due to the performance of the initially trained classifier being already outstanding (98% as shown in the table), the performance improvement is limited. Compared to the state-of-the-art methods, our method outperforms under the settings of different numbers of labeled data in AG News, IMDB, and DBpedia. On Yahoo! Answer, it also performs better than the state-of-the-art methods except under 10 labeled data. These results verify the effectiveness of our PGPL model for the SSTC task.
As some models use a different number of labeled and unlabeled texts during training, we con-
2https://github.com/pytorch/fairseq
duct experiments using their original settings and compare them as in Table 3. The compared models includes UST (Mukherjee and Awadallah, 2020), VAMPIRE (Gururangan et al., 2019), CEST (Tsai et al., 2022), SALNet (Lee et al., 2021), FLiText (Liu et al., 2021), SAT (Chen et al., 2022) and Delta-training (Jo and Cinarel, 2019). For consistency, we take the results from the original papers and round some of the results to one decimal place. The results show that our model outperforms other SOTA models under the same experimental set-
ting keeps its superiority when applied to different experimental settings.

heading: 5.5 Impact of Labeled and Unlabeled Data Distribution
text: In order to evaluate our results more accurately, we performed four experiments with different constraints for further comparison: (a) keep the original proportion of data with balanced labeled data balance and imbalanced unlabeled data; (b) keep the original proportion of balanced unlabeled data, but perform a random imbalanced operation on labeled data; (c) keep the original proportion of balanced labeled data, but perform a random imbalanced operation on unlabeled data; (d) perform a random imbalanced operation on labeled data and unlabeled data. The balance here means that the number of data in each class is the same. In the imbalanced setting, for convenience, the number of half classes is twice that of the other half. The experimental result is averaged after three runs on the verification set.
From Table 4, we find UDA show the worst result under the condition (c) and Mixtext under the condition (d) that have inconsistent distribution proportions though the two baselines have simliar results in balanced condition. Our model shows consistently good results in the four settings. Especially when only unlabeled data is imbalanced, the result is improved from 57.9% to 67.8% compared with UDA. In summary, under four different data distribution settings, our model is superior to
other baselines, indicating that our model is more capable of complex data distribution scenarios.

heading: 5.6 Qualitative Evaluation
text: To investigate whether the PAC module helps to separate the clusters of different classes, we compare the visualized embeddings of full PGPL and PGPL without PAC model in Figure 3.
It is observed from the visualized results that when PGPL is implemented without PAC module (e.g., Figure 3 (a)), the boundaries of different clusters are unclear. Concretely, the clusters of different classes are separated to some extent, but not as distinct as those of full PGPL with PAC implemented. This suggests that the PAC module successfully separates the clusters of different classes and reduces the chance of data instances lying near the class boundaries.

heading: 5.7 Class Significance Evaluation
text: It is observed from Table 5 that PGPL with PGP applied improves the performance on all individual classes, especially in World class and Sports class.

heading: 5.8 Ablation Study
text: In order to investigate the impact of each component in the model, we conduct variants on PGPL.
Table 6 shows the results of PGPL by removing each component at a time, averaged after three runs on valid set with 10 labeled data and 5,000 unlabeled data per class. It is observed that both the PAC and PGP individually improve the final model performance, and the combination of PAC and PGP achieves the best performance. TSA module is also advantageous for improving the performance, while applying to DBpedia is an exception because its initial classifier is already outstanding as explained previously. This confirms the effectiveness of our proposed approach.

heading: 5.9 Stability Evalution
text: To investigate the efficiency of different models, we illustrate their training progresses on AG News and Yahoo! Answer datasets. From the results on the AG News dataset shown in Figure 4 (a), we observe that the model w/o PGP performs very poorly at the early stage of training but converges at about the 40th step. That is, the model w/o PAC shows a good start but continues fluctuating, and it cannot reach the convergence even after 100 training steps. The performance at early stage of PGPL is good and the training progress is also stable.
The training process on Yahoo! Answer dataset shown in Figure 4 (b) shows the similar tendency with the results in Figure 4 (a), except that the model w/o PGP and the full PGPL model show similar training stability and convergence speed. These results suggest that the Prototypical Cluster
Separation module is an indispensable component in keeping the training stable and guaranteeing the convergence speed of the PGPL model. They also indicate that including both PAC and PGP modules not only improves the test accuracy of the model but also improves its training stability and training speed. Lastly, we conclude that including PAC in the early training stage provides more reliable pseudo-labeled texts and helps the latter training stage be more stable.

heading: 6 Conclusion
text: In this paper, we propose a semi-supervised model PGPL for SSTC task that applies PAC and PGP strategies. Specifically, PAC constructs prototypes to cluster text representations belongs to the same class by forcing them to be high-density distributed, thus alleviating the underfitting near decision boundary problem. PGP selects reliable pseudo-labeled data nearby prototypes to address the training bias from the imbalanced data. Experimental results demonstrate PGPL is effective and outperforms previous state-of-the-art methods.
Limitations
Although PGPL is proven to be effective according to our intensive experiments, the current design of the PGP module may not be optimal and could be improved in the future. Specifically, on the one hand, to choose the proper number of nearest text instances kc for each class c, we need to obtain the statistic values µyt , µ y <t and computed values µ c t , µc<t, γt for each dataset, which makes it somewhat not easy to scale to many different datasets. On the other hand, we heuristically filter pseudo-labeled texts by the top-k selection, which may also be improved with more systematical approach.

heading: Acknowledgements
text: This work is supported partly by the National Key R&D Program of China under Grant 2021ZD0110700, partly by the National Natural Science Foundation of China (No.62106059), partly by the Fundamental Research Funds for the Central Universities, and partly by the State Key Laboratory of Software Development Environment.

heading: A Hyper-parameter Settings
text: For reproduction, we report our hyper-parameter settings in Table 8. The initial learning rate is tuned in [1e−6, 1e−5] for BERT parameters and [1e−4, 1e−3] for other parameters. The threshold β2 and threshold λ are tuned in [0.1, 1]. Note that the hyper-parameter settings are tuned on the validation data by grid search with 3 trials.

heading: B Evaluation Results with other pre-trained models
text: In order to evaluate our model on different pretraining language models, we conduct three different experiments on supervised and semi-supervised settings: (1) supervised learning with all labels; (2) supervised learning with 10 labels; (3) and seimisupervised with 10 labels. The experimental results are shown in Table 7. It is observed that all results with RoBERTa is better that those with BERT. In semi-supervised setting, PGPL is better than supervised setting with 10 available labels, and also not far behind the results of full supervised setting with
all labels. In other words, PGPL is very efficient when exploiting few labeled data and achieves almost similar results to the supervised setting that fully exploit all labeled data. This indicates that PGPL is more practical in real-world scenarios where annotated data is limited.

heading: C Evaluation Results When Varying Number of Unlabeled Data
text: We also conduct experiments to evaluate our model performances with fixed number of labeled data and different number of unlabeled data. Specifically, the former uses 10 labels and the latter uses between 2,000 and 10,000 labels on AG News and Yahoo! Answer. The results in Figure 5 show that with more unlabeled data, the accuracy increases significantly on both datasets, which further verify the effectiveness of utilizing unlabeled data.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: A1. Did you describe the limitations of your work?
Left blank.
A2. Did you discuss any potential risks of your work? Left blank.
A3. Do the abstract and introduction summarize the paper’s main claims? Left blank.
A4. Have you used AI writing assistants when working on this paper? Left blank.

heading: B Did you use or create scientific artifacts?
text: Left blank.
B1. Did you cite the creators of artifacts you used? Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Left blank.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.

heading: C Did you run computational experiments?
text: Left blank.
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Left blank.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Left blank.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Left blank.

heading: D Did you use human annotators (e.g., crowdworkers) or research with human participants?
text: Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Left blank.

heading: Future Work
text: Limitations
Although PGPL is proven to be effective according to our intensive experiments, the current design of the PGP module may not be optimal and could be improved in the future. Specifically, on the one hand, to choose the proper number of nearest text instances kc for each class c, we need to obtain the statistic values µyt , µ y <t and computed values µ c t , µc<t, γt for each dataset, which makes it somewhat not easy to scale to many different datasets. On the other hand, we heuristically filter pseudo-labeled texts by the top-k selection, which may also be improved with more systematical approach