SELP: A Semantically-Driven Approach for Separated and Accurate Class Prototypes in Few-Shot Text Classification (2024)

Abstract: The meta-learning paradigm has demonstrated significant effectiveness in few-shot text classification. Currently, numerous efforts are grounded in metric-based learning, utilizing textual feature vectors for classification, with a common emphasis on enlarging inter-class distances to achieve improved classification effectiveness. However, many methods predominantly focus on enhancing the separation of prototypes without taking the semantic relationships between prototypes and class clusters into consideration. This oversight results in incomplete and inaccurate encoding of prototypes within the semantic space, affecting the generality of the learned metric space. In this paper, we propose the utilization of Semantically Enhanced Labels for calibrating class Prototypes (SELP), thereby obtaining prototypes that are more separated and semantically accurate. Additionally, we have devised a center loss to enhance intra-class compactness, coupled with the introduction of a simulated label distribution method to address the overfitting problem. Extensive experiments on eight few-shot text classification datasets show that the proposed method outperforms baselines significantly. Our code is available at https: //github.com/tttyyyzzz-zty/SELP.git.

heading: 1 Introduction
text: Text classification is a crucial and foundational task in natural language understanding, widely employed across various domains such as intent recognition (Dopierre et al., 2021) and sentiment analysis (Kumar and Abirami, 2021). The application of deep learning approaches to text classification tasks, combined with extensive supervised training on vast data, has demonstrated substantial efficacy in achieving superior performance in text classification endeavors (Devlin et al., 2019; Raffel et al., 2020; Song et al., 2020). However, the collection
*Corresponding author.
of such a substantial volume of annotated data is time-consuming and laborious, rendering it impractical across numerous real-world domains, thereby motivating few-shot text classification.
There are several methods proposed to solve few-shot text classification tasks. Fine-tuning based methods (Howard and Ruder, 2018; Shen et al., 2021) often use a pre-trained language model (PLM) and then fine-tune the model to the downstream task. A variant of this approach is promptbased methods (Li and Liang, 2021; Schick and Schütze, 2021), which typically transform text classification tasks into closed-text formats to bridge the gap between pre-train and downstream tasks to better utilize the capabilities of the pre-trained model. However, transforming the task into a fillin-the-blank format entails inherent limitations, demanding meticulous and precise task design, which might lack applicability across various real-world tasks. Data augmentation based methods (Liu et al., 2019b; Dopierre et al., 2021) aim to utilize auxiliary data or information for data augmentation or feature augmentation of few-shot datasets, but they may introduce new noisy data, which are more dependent on prior knowledge, and may lead to feature loss. Meta-learning based methods (Lei et al., 2023; Chen et al., 2022) aim to give the model the ability to quickly generalize to novel classes by learning on different simulated small episodes (tasks). Meta-learning based methods have shown promising results in few-shot tasks and are considered a highly promising methodology.
Although all of the above methods achieve good performance, there are still some problems with the current methods. For methods that utilize textual feature vectors for classification, their performance is highly dependent on the inter- and intra-class variance in the query set. The key to solving this problem is to encourage greater intra-class compactness and inter-class separability explicitly, and there are already some works to do so. ContrastNet
9732
(Chen et al., 2022) and TART (Lei et al., 2023) employ contrastive learning methods to promote mutual separation among prototypes. However, these methods primarily emphasize prototypes that are more separated, neglecting the advanced semantic relationships between prototypes and class clusters. In some cases, the computed prototypes may not effectively represent their corresponding classes. This leads to inaccuracies in encoding prototypes within the semantic space, consequently affecting the generality of the learned metric space and diminishing model generalization. We propose to address the aforementioned issues by employing labels to calibrate prototypes, aiming to obtain separable and more accurate prototypes. While some existing work utilizes labels for classification (Luo et al., 2021; Du et al., 2023), they do not manipulate the prototypes. Moreover, some methods resort to leveraging external knowledge bases to augment label semantics due to the limited information contained in labels (Zhang et al., 2022). We contend that this approach is intricate, and the obtained labels lack genuine instance-based information, resulting in limited generalization capability.
In this study, we introduce a simple yet effective approach that utilizes a prompt pool to enrich the semantic content of labels. This method ensures that labels encapsulate richer information about the underlying semantics of instances within their associated classes. Subsequently, we utilize these enriched labels to calibrate class prototypes, encouraging increased distinctiveness among prototypes and bolstering their semantic accuracy. Additionally, we introduce a center loss to enhance intra-class compactness and a simulated label distribution method to mitigate the overfitting problem.
Figure 1 illustrates the comparison between our approach and two strong baseline methods. It can be observed that our method effectively pulls back the prototypes to the locations where class clusters aggregate, achieving better intra-class compactness.

heading: 2 Related Work

heading: 2.1 Fine-Tuning Based Methods
text: Fine-tuning based methods typically use pretrained language models (PLMs) and then fine-tune them on specific downstream tasks to obtain models adapted to the downstream task. Some methods (Howard and Ruder, 2018; Shen et al., 2021) aim to preserve the transferability of the model by applying distinct learning rates to each layer during training. However, fine-tuning based methods suffer from the overfitting problem due to the scarcity of training data. Recently, prompt-based approaches have achieved promising results. PET (Schick and Schütze, 2021) models classification problems as fill-in-blank problems, bridging the gap between upstream and downstream tasks. LM-BFF (Gao et al., 2021) uses the T5 (Raffel et al., 2020) for automated template generation, and in turn combines examples with the current text, enriching the contextual information. However, transforming the task into a fill-in-the-blank format entails inherent limitations, demanding meticulous and precise task design, which might lack applicability across various real-world tasks.

heading: 2.2 Data Augmentation Based Methods
text: The core idea of data augmentation based approaches is to utilize auxiliary data or information for data augmentation or feature augmentation of
few-shot datasets. Dopierre et al. (2021) propose a short text paraphrase model, which produces different paraphrases of the original text as data augmentation. TPN (Liu et al., 2019b) employs a transductive approach by constructing an undirected graph that integrates all unlabeled and labeled data, obtaining labels for all unlabeled data through label propagation. Way-DE (Liu et al., 2023) assumes a Gaussian distribution for each class and utilizes the original support set along with the nearest minority query samples to estimate the mean and covariance. Subsequently, it augments labeled samples by sampling from the estimated distribution. While data augmentation remains a reliable method for addressing few-shot problems, it may inadvertently introduce new noisy data, become more reliant on prior knowledge, and potentially lead to feature loss.

heading: 2.3 Meta-Learning Based Methods
text: Meta-learning aims to enhance the multi-tasking generalization of the model by sampling episodes on the seen classes so that it can be quickly adapted to novel classes. Existing meta-learning based methods can be divided into three types. (1) Optimization-based methods, such as MAML (Finn et al., 2017) and MAML++ (Antoniou et al., 2019), aim to learn a good model initialization parameter that can be quickly adapted to a new task within a few steps of gradient updating. (2) Modelbased methods, such as MANN (Santoro et al., 2016) and Meta-ticket (Chijiwa et al., 2022), develop specific model architectures that enable rapid adaptation to novel tasks. (3) Metric-based methods like prototypical network (Snell et al., 2017) and induction network (Geng et al., 2019), utilize a metric function to calculate the distance or similarity between different samples, which in turn determines the category of the sample to be predicted. Current metric-based methods focus on generating separable feature representations, ContrastNet (Chen et al., 2022) through supervised contrastive learning and two unsupervised contrastive learning approaches at the task level and instance level, achieves more discriminative prototype representations and alleviates the issue of overfitting. TART (Lei et al., 2023) transforms the class prototypes to per-class fixed reference points in task-adaptive metric spaces and uses a discriminative reference regularization to further maximize divergence between transformed prototypes. Meta-learning stands as a promising approach for
addressing few-shot tasks.

heading: 3 Model

heading: 3.1 Problem Formulation
text: In this paper, we follow the traditional N-way Kshot setting. Specifically, let Ctrain, Cval, Ctest denote the disjoint set of training classes, validation classes and test classes, and they have no overlapping classes, i.e., Ctrain ⋂ Cval ⋂ Ctest = ∅.
In the training phase, we construct training episodes Ntrain from Ctrain, and each task contains a support set S and a query set Q. We randomly select N classes of K samples each to form the support set, i.e., S = {(xi, yi)}N×Ki=1 , where xi is a data sample, yi is the class label. The query set consists of a portion of the remaining samples from these N classes, i.e., Q = {(xj , yj)}N×qj=1 , where q is the number of queries.
In the validation and testing phases, we use the same approach to construct Nval and Ntest from Cval and Ctest, respectively. But in the testing phase, as the labels of queries are unknown in the testing stage, the query set in a test task can be represented as Q = {xj}N×qj=1 . A meta-learner is trained on such episodes that attempt to classify the texts in the query set Q on the basis of few labeled texts in the support set S.

heading: 3.2 Overview
text: Our approach employs BERT (Devlin et al., 2019) as a text encoder and utilizes a prompt pool to enhance the semantic content of labels. Subsequently, the enhanced labels are used to calibrate text prototypes, aiming for more distinctive and semantically accurate text representations. Additionally, we design a center loss to enhance intra-class compactness and introduce a label distribution estimation method to mitigate overfitting issues. The overall model structure is shown in Figure 2. All notations in Figure 2 will be defined in the rest of this section.

heading: 3.3 Class-Dependent Prompt
text: Prompt Pool In contrast to using only a single initialization for the prompt, Wang et al. (2022) propose to learn a prompt pool. Prompt pool can extract more knowledge about the task than a single prompt and is more suitable for hard and complex tasks. The prompt pool comprises T trainable prompts, which are dynamically updated across tasks to acquire transferable metaknowledge. Specifically, we define the prompt pool
as follows:
V = {V1,V2, ...,VT },Vi ∈ RL×dh , (1)
where T is the number of prompts in the pool, L is the length of each prompt and dh is the word embedding dimension. Each prompt is attached to a learnable key kj :
K = {k1,k2, ...,kT },ki ∈ Rdh . (2)
Prompt Enhanced Labels Here, we are not using the prompt pool to build prompt templates, but rather to use the prompt pool to enhance labels. Prompt pool is able to learn meta-knowledge that is transferable across tasks, in each episode, we can compute a prompt for each class name through the attention mechanism. Specifically, given a class c, the weight between class name ac and prompt Vj is computed as:
wcj = k⊤j a cls c∑T
j′=1 k ⊤ j′ aclsc
, (3)
where ac = E(ac) ∈ RLc×dh is the label name features of class c, obtained from the pre-trained language model(i.e., E(·)). And aclsc ∈ Rdh is a
standalone token embedding encompassing the entire semantic label (we employ the CLS token from BERT here). Then the class-dependent prompt for class c is generated by weighted all the prompt values:
vc = T∑
j=1
wcjVj . (4)
Because the prompt for each class is computed based on the class name, it is task-independent, and this design also allows for quick adaptation and computation for novel classes. Following the computation of a class-specific prompt, we concatenate the class name with the generated prompt to obtain a new label name tc = [vc;ac].
It is worth noting that, to ensure the enhanced label representations and text representations remain in the same space, we further pass their representations through an additional encoder, mapping them into a unified space. Specifically, in our implementation, we employ the first 11 layers of BERT as the first text encoder (i.e., E(·) ) and use the last layer of BERT as the second text encoder (i.e., fe(·)). Consequently, we obtain the label representation t′ = fe(t) and the text representation x = fe(E(x)) within the same vector space.
Calibrated Prototype Representation To address the issue of insufficient separation and accuracy of prototypes, we calibrate the prototypes using enhanced labels. Specifically, we first obtain the class prototype pc of each class through the support set
pc = 1
K
∑
(xi,yi)∈Strain I(yi = c)xi, (5)
where xi ∈ Rdh is the text features, I(·) is the indicator function. We calibrate it with semantically enhanced labels to get the final class prototype p′c:
p′c = α× pc + (1− α)× t′c. (6)
Here, α is a hyperparameter ranging between 0 and 1.

heading: 3.4 Center Loss
text: In the previous step, we have achieved improved separation among class prototypes. Now, our goal is to ensure greater compactness within each class. We propose a simple regularized loss:
Lcen = 1
2Nq
Nq∑
i=1
||xi − p′yj ||2, (7)
where Nq is the number of query samples in an episode. With such a simple loss, it is possible to force each sample to move closer to the prototype of the class, making the compactness within each class increase.

heading: 3.5 Label Distribution Estimation
text: Due to the scarcity of training samples, few-shot text classification tasks often suffer from the overfitting problem. One contributing factor to model overfitting is the inadequacy of the current one-hot encoding of labels, which fails to fully capture the relationships between instances and labels. This inadequacy arises from the non-complete independence among labels, and instances may, in practice, be associated with multiple labels. This results in the model exhibiting overconfidence on the seen classes, consequently leading to diminished generalization performance on unseen classes.
Inspired by Guo et al. (2021), we aim to transform the current one-hot encoding, a "hard label," into a "soft label" by modeling the relationships between instances and labels. This approach is employed to facilitate the model in learning as much information as possible from the instances, thereby alleviating overfitting concerns. Specifically, in a
specific episode, for each query sample, we calculate its similarity with each label:
y(c) = softmax(xiT⊤W + b), (8)
where T = [t′1, t ′ 2, ..., t ′ N ] is the set of labels that have been enhanced in this episode. The similarity we calculated is then weighted with the original one-hot label y(t), and after weighting, we get the simulated label distribution (SLD) that we ultimately wish to obtain.
y(s) = softmax(βy(t) + y(c)). (9)
Based on the prototypes obtained from Eq.6, we calculate the distance between each sample and the prototypes as the predicted label of the sample:
p(y = c|xq) = exp (−d(xq,p′c))∑N i=1 exp (−d(xq,p′i)) , (10)
y(p) = softmax([p1, p2, ..., pN ]), (11)
where xq is a query instance. We use the Kullback–Leibler divergence as the loss function to measure the difference between y(p) and y(s).
Llcm = KL-divergence(y(s), y(p))
= N∑
i=1
y (s) i log(
y (s) i y (p) i ) . (12)

heading: 3.6 Objective and Prediction
text: Overall Objective During the training phase, we amalgamate the computed losses Lcen in Eq.7 and Llcm in Eq.12. Given that different losses may exhibit varying scales, and manually adjusting weights can be a challenging and expensive process, we employ uncertainty weights (Kendall et al., 2018) to automatically compute the weight for each loss. The overall objective is:
L = 1 2σ21 Lcen + 1 2σ22 Llcm + log σ1σ2. (13)
Testing In the testing phase, given an N -way K-shot task, we first compute the corresponding prompt for each label, then compute prototypes for each class using the samples in the support set, then calibrate the resulting prototypes with the labels, and finally compute the distance from the samples in the query set to each prototype for classification.

heading: 4 Experiments

heading: 4.1 Datasets
text: In line with previous work (Liu et al., 2023), our experiments will be conducted on eight commonly used datasets, including four intent detection datasets: Banking77, HWU64, Clinc150, and Liu57, and four news or review classification datasets: HuffPost, Amazon, Reuters, and 20News. The detailed statistics for all datasets are summarized in Table 1. We provide a brief introduction to the dataset: HuffPost (Bao et al., 2020) consists of news headlines published on HuffPost between 2012 and 2018. Amazon (He and McAuley, 2016) consists of 142.8 million user reviews across 24 product categories. Following (Han et al., 2021), we use a subset, selecting 1,000 reviews for each category. Reuters (Bao et al., 2020) is collected shorter Reuters articles in 1987. Following (Bao et al., 2020), we employ a subset of 31 classes. 20News (Lang, 1995) is comprised of informal discourse from news discussion forums, covers 18828 documents under 20 topics, is a subset of 20 news groups. Banking77 (Casanueva et al., 2020) is a finegrained intent classification dataset specific to banking domain. HWU64 (Liu et al., 2019a) is a fine-grained intent classification dataset. Liu57 (Liu et al., 2019a) is a highly imbalanced intent classification dataset collected on Amazon Mechanical Turkcollected from Amazon Mechanical Turk. Clinc150 (Larson et al., 2019) comprises 150 intents and 23,700 examples spanning 10 domains.

heading: 4.2 Baselines
text: We compare the proposed few-shot text classification models with the following baselines: Prototypical Network (Snell et al., 2017) learns class prototype representations and employs nearest-neighbor classification, emphasizing instance similarity to prototypes. MAML (Finn et al., 2017) optimizes for rapid adaptation to new tasks through iterative training across multiple tasks, acquiring generic initial parameters for quick learning and generalization across diverse tasks. Induction Network (Geng et al., 2019) uses dynamic routing to represent and condense samples within categories into class-level representations, facilitating query sample classification. HATT (Gao et al., 2019) utilizes a hybrid attention mechanism, encompassing both instance-level and feature-level attention, to enhance robustness and expedite the iteration speed of the model. DS-FSL (Bao et al., 2020) focuses on learning the relationship between the importance of words and distributional signatures, thereby weighting to obtain a more refined sample representation. MLADA (Han et al., 2021) is a meta-learning framework integrated with an adversarial domain adaptation network, aiming to improve the adaptive ability of the model and generate high-quality text embedding for new classes. ContrastNet (Chen et al., 2022) through supervised contrastive learning and two unsupervised contrastive learning approaches at the task level and instance level, achieved more discriminative prototype representations and alleviated the issue of overfitting. TPN (Liu et al., 2019b) employs a transductive approach by constructing an undirected graph that integrates all unlabeled and labeled data, obtaining labels for all unlabeled data through label propaga-
tion. PROTAUGMENT (Dopierre et al., 2021) is a data augmentation technique that uses a model to generate paraphrases of short texts. It applies an unsupervised loss at the instance level on the vanilla prototypical network (Snell et al., 2017). PROTAUGMENT (unigram) and PROTAUGMENT (bigram) employ different strategies for word paraphrasing. TART (Lei et al., 2023) transform the class prototypes to per-class fixed reference points in taskadaptive metric spaces and use a discriminative reference regularization to further maximize divergence between transformed prototypes. Way-DE (Liu et al., 2023) assumes a Gaussian distribution for each class and utilizes the original support set along with the nearest minority query samples to estimate the mean and covariance. Subsequently, it augments labeled samples by sampling from the estimated distribution.

heading: 4.3 Implementation Details
text: Evaluation Metric We follow Liu et al. (2023) to use accuracy to assess the performance of our model. The datasets used in our experiments, as provided by Chen et al. (2022), consist of five random class partitions for each dataset. All reported results are averages obtained across these five partitions.
Parameter Settings We follow Liu et al. (2023) to conduct experiments on 5-way 1-shot and 5-shot settings. Across the news or review classification datasets, we employed bert-base-uncased model as the feature extractor. Our reported average accuracy is based on 1000 episodes sampled from the test set, with each episode comprising 25 query instances. Across the intent detection datasets, we employed a further pre-trained BERT model provided in Dopierre et al. (2021) as the feature extractor. Our reported average accuracy is based on 600 episodes sampled from the test set, with each episode comprising 5 query instances. We set the number of prompts T , to 8, and the length of the prompt is chosen from {8, 16} using the validation set. In the simulation of label distribution, we set β to 4.0. We optimize the models using AdamW with an initialized learning rate of 2e-5. In the prototype calibration stage, we opt for distinct values of α for each dataset from {0.5,0.6,0.7,0.8} based on the performance of the validation set.

heading: 4.4 Main Results
text: Tables 2 and 3 report the experimental results for the news or review classification task and the intent detection task. Most baseline results are taken from Liu et al. (2023). The results for TART in Table 2 and Table 3 are obtained from our re-run of their
experiment. The best results are highlighted in bold.
From Table 2, it is evident that our approach achieves state-of-the-art performance on most datasets, particularly excelling in the 1-shot setting with an average improvement of 6.4% over other baselines. In the 1-shot setting of the HuffPost dataset, our method outperforms the best baseline by 14.2%. The reason behind this lies in the 1- shot scenario, where prototypes, being computed from a single sample, are more prone to deviate from class clusters. Our method effectively pulls back these deviated prototypes, leading to superior performance. This underscores the superiority of our approach in handling few-shot tasks. Besides, we notice that the performance gains for 1-shot intent detection are smaller than the gains for 1- shot text classification. We believe there are two main reasons for that: (1) The baselines of the intent detection datasets are very high, and there is not much room for improvement. (2) The texts in the intent datasets are shorter and provide limited information, which is a limitation for our method.
As depicted in Table 3, our method demonstrates consistent effectiveness, outperforming baselines on the majority of datasets. Similarly, our approach exhibits more pronounced improvements in the 1- shot setting, 1.6% in 1-shot on Banking77, and 2.7% in 1-shot on HWU64, respectively This further validates the effectiveness of our method for both long and short texts.

heading: 4.5 Visualization
text: In Figure 1, we employ t-SNE to visualize the query and prototype representations generated by ContrastNet, TART, and our method. We sample 5 classes from the test set of HWU64, sampling one prototype and 25 samples for each class. From Figure 1(a), ContrastNet roughly distinguishes class textual representations. However, the prototype representations sometimes significantly deviate from the class clusters, leading to a substantial impact
on the classification accuracy of ContrastNet. As illustrated in Figure 1(b), TART maps the representations to another space, resulting in a different distribution compared to the other two methods. It can be observed that, although TART attempts to separate the prototypes as much as possible, the query samples are intermingled without clear boundaries. In Figure 1(c), our approach demonstrates that similar classes still cluster together. However, class prototypes are drawn back to their respective semantic regions through label calibration. Furthermore, the intra-class cohesion is relatively enhanced, and inter-class boundaries are clearer, affirming the effectiveness of our method.

heading: 4.6 Ablation Study
text: To demonstrate the impact of each component of the model, we conduct ablation experiments on the HuffPost and HWU64 datasets, as shown in Table 4.
The SELP -Llcm donates that we do not apply Llcm (Eq.12) and directly use one-hot label and cross-entropy loss. The decrease in the model’s performance indicates the effectiveness of the designed LCM loss.
The SELP -Lcen donates that we do not apply Lcen (Eq.7) and solely employ Llcm as the loss function for model updates. There is also a decline in the effectiveness of the model.
The SELP -PromptPool donates that we excluding the prompt pool, we directly employ the embeddings of the raw labels to calibrate the prototypes and compute the overall loss for model updates. The model exhibits a significant performance decline, indicating that the prompt pool has learned meta-knowledge, which enhances the representation of labels.
When removing the label information (SELPLabel), the model regresses to the original prototype network. And the model performs the worst, indicating the strong guidance role of label information in classification.

heading: 5 Conclusion
text: In this paper, we employ a prompt pool to compute a distinctive prompt for each label. This prompt encompasses learned transferable meta-knowledge, providing additional semantic information related to instances. Subsequently, the prototypes are calibrated using these enhanced labels, resulting in prototypes that are more separated, semantically
accurate, and closer to their respective class clusters. Additionally, we introduce a straightforward center loss to enhance intra-class compactness and apply a simulated label distribution method to mitigate overfitting issues. Experimental results affirm the effectiveness of our proposed approach.
Limitations and Potential Risks
Limitations In this paper, we employed BERT as the text encoder. While our proposed approach is applicable to any PLM that provides text representations, we did not conduct experiments on other PLMs due to time and efficiency constraints. Additionally, our method relies on meta-learning and therefore requires at least one available training episode. In future work, we aim to further investigate the application of labels in the context of small-sample and zero-shot scenarios. Potential Risks Our research is dedicated to investigating how to enhance natural language understanding under low-resource conditions, with a focus on improving the performance of text classification. Our efforts contribute to an uplift in text classification efficiency, with no inherent risks posed to society or individuals.

heading: Acknowledgment
text: This work was supported by National Natural Science Foundation of China (No. 62106035, 62206038), and Chunhui Project Foundation of the Education Department of China (No. HZKY20220419).

heading: Future Work
text: In future work, we aim to further investigate the application of labels in the context of small-sample and zero-shot scenarios. Potential Risks Our research is dedicated to investigating how to enhance natural language understanding under low-resource conditions, with a focus on improving the performance of text classification. Our efforts contribute to an uplift in text classification efficiency, with no inherent risks posed to society or individuals