DEIM Forum 2023 1a-1-4

Soft and Hard Prompting for Document Classification with Only Label Names
Luyao WANG†

Zhewei XU‡

andMizuho IWAIHARA‡

Graduate School of Information, Production and Systems, Waseda University
2-7 Hibikino, Wakamatsu-ku, Kitakyushu, Fukuoka 808-0135 Japan
E-mail: †luyao@fuji.waseda.jp, ‡xuzhewei@toki.waseda.jp, ‡iwaihara@waseda.jp
Abstract The existing prompt-based tuning methods can tune discrete or continuous prompts with a frozen language model,
and achieve performance comparable to fine-tuning on many tasks. However, it is often difficult to achieve good results without
any annotated data. Especially for continuous prompt optimization, it is usually necessary to provide a certain amount of labeleddata to train a prompt vector. Therefore, we propose a method that combines soft and hard prompts for text classification tasks.
First, we construct discrete prompts and a verbalizer to obtain initial prediction results by the masked language model. Then apart of the labelled data with high confidence are used as pseudo-labeled training samples to optimize the continuous prompt
vector. Finally, a classification model can be obtained by self training. We find that the ideas of prompting can be applied to
weakly supervised learning methods, and it is worth exploring in the future research.
Keyword Document classification, continuous prompt, weakly supervised learning, pretrained language model, selfoptimizing
the topic of a sentence x: “Amazing film!” into the “GOOD”

1. Introduction
Recent years pretrained language models (PLMs) havecategory, we wrap it into a template: “x This film isbeen widely applied to various natural language processing[MASK]” and the generated words in “[MASK]” token aretasks due to their powerful language understanding ability.

mapped into categories via the verbalizer. However, it was

To explore the principles of such effectiveness of PLMs,

found that even small differences in the hand -constructed

researchersandtemplates could have a significant impact on the results .

suggested that PLMs have obtained rich knowledge during[17] have attempted to automatically search for discrete

pre-training [2][14]. As a result, more and more attention

templates, but this approach tends to fall into local optima.

is being paid to how to better utilize and maximize the

To overcome this dilemma, a series of continuous prompts

potential of language models. A common method to achieve

methods [6][8][9] have been proposed. The prefix-tuning

this goal is fine-tuning [3], in which an additional classifier[6] optimizes consecutive word embeddings as prefixes tois added to the top of a PLM and the model is further

indicate text generation task s, with the parameters of the

trained under its classification objectives. Fine-tuning has

pre-trained language model frozen. It shows that only ayieldedsmall amount (about 0.1%) of optimization parameters are

haveconducted

satisfactory

results

extensiveonstudiessupervised

tasks.

Nevertheless, applying fine-tuning in few-shot learning [1]needed to achieve good results.

and zero-shot learning [19] scenarios remains challenging

When a prompt-based approach applied to a class name-because additional classifiers require sufficient training

only text classification task, we have no labeled text thatsamples for fine-tuning.

can be used to optimize continuous prompts, and theWith the advent of GPT-3 [1], researchers have also

template and verbalizer construction may produce moreincreasingly focused on prompt-based learning approaches.

bias in the results if we use discrete prompts. Therefore,

The basic idea is to link downstream tasks to pre-trained

we propose a new method combining hard prompts and softprocess. A series of research using prompts [7][15]prompt-tuning (HSPT). As shown in Figure 1, Ourdemonstrates that such discrete or continuous prompts

approach can be divided into three stages. Firstly, weshow better performance on zero-shot and few-shot tasks,

obtain highly-credible pseudo-label data by using a hardeven on small-scale models. After the Pattern-Verbalizer-

prompt and the masked language model, and then it is used

Pair (PVP) component is proposed, a routine flow of using

as training samples to optimize the continuous prompt s indiscrete prompts is formed. That is, we first put the input

the second stage. Finally, we can obtain the final

sentence into a natural language template , then let the PLM

classification model after several iterations of optimization.

make the masked language prediction, and map the result

We conduct experiments

to the class using the verbalizer. For example, to classify

Figure 1. Overview of the prediction process using Pattern-Verbalizer-Pair and discrete prompt
on four datasets without any labeled data. Experimental

relation extraction, and knowledge probing, etc. However,results demonstrate that our design is feasible and effective,

the hard prompt is not precise enough to be used as a basis

which achieves good results on most of the datasets , so thisfor weakly supervised classification, and the soft prompt

direction is worth exploring.

can not be applied directly to this task either, b ecause we
do not have any ground-truth labeled data in our setting,

2. Related Work

there is a lack of material for training soft prompt

2.1. Prompt-tuning

parameters. As a result, we consider the way of combining

Prompt-tuning originated with the emergence of thethese two kinds of prompts to solve the problem, by first

GPT-3 [1] model, with arguments that large-scale language

generating pseudo-labels through hard prompts, and thenmodels can maximize their reasoning and understanding

extracting the parts of them with high confidence to train

capabilities with appropriate templates. GPT-3 pioneers thesoft prompts.

concept of in-context learning, enabling few-shot or zeroshot learning without modifying the model. Later, Schick2.2. Class Name-Only Text Classification

et al. [15] demonstrate that this approach is also feasible

Class name-only text classification means that only label

on small-scale language models like BERT [3]. They

surface names or limited word -level descriptions of eachattempt to convert all classification tasks into cloze

category can be used for classi fication. Recently, a number

questions consistent with the masked language modelof(MLM) and designs an important component, Pattern-classifiers in an weakly supervised way. They generate a

Verbalizer-Pair (PVP). Based on this framework, current

set of documents with pseudo labels to train a supervised

research has focused on how to select or constructmodel over them. LOTClass [12] utilizes BERT to query

appropriate templates and verbalizers. A simple approach

replacements for label names to obtain candidate words foris to design templates manually based on the nature of thecategories, and performs self-training on unlabeled datatask and prior knowledge, but it is found that the choice ofafter fine-tuning the PLM. X-Class [14] leverages BERT to

hard prompts has a very large impact on the pre-trained

represent documents and labels, and generates document -model. To solve this problem, soft prompt is proposedclass pairs by clustering to train a classifier. Compared[6][8][9], introducing new parameters into the model towith previous work, our proposed HSPT method obtainmake prompt generation a task for the machine to learn .

supervision data by MLM with discrete prompts, instead of

The essence of prompt-tuning is to reformat the task so as

utilizing a set of category vocabulary, or calculating theto cater to the performance of the larg e-scale model.

similarity between classes and texts. We further introduce

Prompt-tuning has been applied to a large variety of tasks

continuous prompts to obtain classification results bysuch as text classification, natural language understanding,

prompt-tuning rather than fine-tuning, which requires lessresearches[11][12][15][16]trainedneuraltext

Figure 2. Overview of the prediction process using Pattern-Verbalizer-Pair and discrete prompt
training parameters.

Finally, we assign the category with the highest probability
to text 𝑥 as a pseudo label.

3. Methodology

Verbalizer Construction The verbalizer is an

3.1. Hard Prompt

important part of the PVP component, mapping a set of

We denote 𝑀 as a pre-trained language model. In text

predicted words into the label space. A common idea is toclassification task, we suppose to classify the input text set

look for category related words, so we u tilize the category

𝑋 = {𝑥0 , 𝑥1 , … , 𝑥𝑛 } into class set 𝑌 = {𝑦0 , 𝑦1 , … , 𝑦𝑚 }. Discrete

vocabulary built in LOTClass [12], which is obtained byprompting aims to formalize the classification task into ausing MLM to predict the position of category words andmasked language modeling problem in cloze style. For

rank them by frequency of occurrence. We take the top 10

example, assuming that we need to classify the text 𝑥 =words of each class in the category vocabulary and add“At the beach volleyball, the 2004 Olympics is a foot -them to the verbalizer. The procedure of predicting masked

stomping success.” into a label from set {politics, sports,words based on the context is not a single -choice process,business, technology}. We can add a template and changethat is, there is no standard correct answer, but abundant

the raw text into:words may fit this context. Therefore, we also introduce
WordNet [13] as an external structured knowledge base to𝑥𝑡 = [CLS] 𝑥 𝑇ℎ𝑖𝑠 𝑡𝑜𝑝𝑖𝑐 𝑖𝑠 𝑎𝑏𝑜𝑢𝑡 [MASK]

expand the verbalizer with comprehensive label words. We

Taking it as input, MLM can generate the probability ofwords as verbalizer words, ranked by word similarities.

generate synset for each class, and choose another top 10

each word 𝑣 in the vocabulary appearing at the [MASK]

Generating Highly Confident Data After obtaining

position. Then we design a verbalizer, which consists of athe classification result for each document, we sort all texts

set of related words from the vocabulary of each class. We

by their maximum class probabilities. In order to optimize

define a label word set 𝑉 as the words that are used by thethe soft prompt, we tend to choose samples with highverbalizer. As Figure 2 shows, the verbalizer can bequality in the first stage, where a large amount is not so

consideredthenecessary. From this perspective, we select the top α% ofprobabilities of predicted words into the probabilities ofsorted results and add them to the high confidence set. In

categories. Then the probability of label 𝑦 for sentence 𝑥𝑡

the experiments, we set α as 20.

asa

mapping

𝑓∶𝑉→𝑌

to

map

can be calculated as

3.2. Soft Prompt-tuning
𝑃(𝑦|𝑥𝑡 ) = 𝑃𝑀 ([MASK] = 𝑣|𝑥𝑡 ) | 𝑣 ∈ 𝑉𝑦 , (1)It is suggested that manually constructed templates and𝐶𝑆𝑖 = 𝑚𝑎𝑥 𝑝𝑖𝑗 , 𝑗 = 1,2, … , 𝑚, (3)verbalizer may create uncertainty into the results. However,continuous prompt learning does not suffer from thisproblem. Therefore, we use high confidence data generated
in the first stage to optimize the continuous prompts toimprove the stability and generalization ability of themodel.
Liu et al. [7] and Lester et al. [5] introduce trainable
continuous prompts as a substitution to natural language
prompts for natural language understanding (NLU) withthe parameters of pretrained language models frozen.
Prompt tuning has been proved to be comparable to fine tuningon

10-billion-parametermodelsonsimpleclassification tasks [5][7]. As Figure 2 shows, continuous
prompts are added as prefixes to the input of each layer.
We

keep

the

parameters

of

the

pre-trained

Figure 3. Overview of the soft prompt-tuning and self-

model

optimizing stage

unchanged and only optimize the prefix vector withpseudo-labeled text data. The work [8] demonstrates that

Then, as illustrated in Figure 3, we sort all documents bythis kind of deep prompt-tuning optimizes more smoothly

the confidence score and use 𝑡𝑜𝑝(𝛼 + 10 ∗ 𝑖)% of them as

and more effective than just adding prompts to thehigh confidence predictions for prompt optimization in theembeddingi-th self-optimizing loop.

layer

like

[9]

for

naturallanguage

understanding (NLU) tasks. Then we utilize the traditional
method for classification through [CLS] tokens, with

4. Experiments

randomly initialized linear head s, rather than the verbalizer.

4.1. Datasets

We use soft prompts to solve the text classification task

We

use

three

benchmarkdatasetsfordocument

without any labeled data. Therefore, compared with theclassification, including three topic datasets AGNews [20],previous work, our training of soft prompts is not based on

DBpedia [4] and 20News and a sentiment dataset IMDB

ground-truth labeled data, but on pseudo -labeled data,[10]. The dataset statistics are shown in Table 1.

which can exploit the potential of prompt-tuning to agreater extent.

4.2. Compared Methods
We compare our method with a wide range of weakly-

3.3. Self-Optimization

supervised methods and also state-of-the-art supervised

Generally, after optimizing the soft prompt using

methods. Fully supervised methods use the entire training

selected pseudo-labelled data with high confidence, theset for model training. Weakly-supervised methods use thepretrained language model can generate more accurate

training set as unlabeled data.

classification results. As a result, we propose to self-

WeSTClass [11]: This method first generates pseudo

optimize the prefix vector on the entire unlabeled data. The

labels for texts that include user-provided candidate words.

process of self-optimization is to iteratively use theAlso, pseudo-labelled documents are used as training data

model’s current predictions with high confidence toto train a neural network. Then it performs a self-training

computeprocess.

atargetdistribution

which

optimizethecontinuous prompt simultaneously. We define 𝑟𝑖𝑗 as thelogits of the LM for document 𝑖 belongs to class 𝑗, andLOTClass [12]: It builds a category vocabulary for eachcategory, by utilizing a pre-trained masked language model

we obtain a probability distribution 𝑃 = [𝑝𝑖1 , 𝑝𝑖2 , … , 𝑝𝑖𝑚 ]to find replaceable words. Then the LM is finetuned

over labels using the softmax function:

through word-level class prediction task, and finallygeneralized classifier is obtained after self-training on[𝑝𝑖1 , 𝑝𝑖2 , … , 𝑝𝑖𝑚 ] = 𝑆𝑜𝑓𝑡𝑚𝑎𝑥([𝑟𝑖1 , 𝑟𝑖2 , … , 𝑟𝑖𝑚 ]), (2)Then the confidence score of 𝑥𝑖 is obtained by:unlabeled data.

X-Class [14]: It leverages BERT representations togenerateclass-oriented

document

presentations,

thengenerates document-class pairs by clustering methods, and

Name

Type

#Class#Train set

20NewsTopic

517,871

AGNewsTopic

4120,000

DBpediaTopic

14560,000

IMDB

Sentiment

2

25,000

Table 1 Dataset statistics

Model

20NewsAGNewsIMDB

DBpedia

WeSTClass71.28/69.90

82.3/82.1

77.4/-

81.42/81.19

LOTClass

73.78/72.53

86.89/86.82

86.5/-

86.66/85.98

X-Class

78.62/77.76

85.74/85.66

82.20/82.18

91.32/91.17

HSPT (proposed)

82.51/81.33

88.66/88.65

89.78/89.73

78.45/78.32

Supervised

93.99/93.99

93.99/93.99

94.55/94.54

98.96/98.96

Table 2 The Micro and Macro-F1 scores of our HSPT method
compared with the baselines and supervised method
then fed pairs to a supervised model to train a text classifier.

and IMDB, but there is still a gap between our result and

BERT [3]: We use the pretrained BERT-base-uncased

the supervised ones. However, our method does notmodel and fine-tune it with the labeled training data for

perform as well on the DBpedia dataset, which may betext classification. It shows an upper bound of weakly

related to the imperfect construction of the verbalizer andsupervised methods.

the imbalance of the data in each category.
The results show that our idea and design are reasonable,

4.3. Experimental Settingsand our model can effectively do text classification under

We use the pre-trained BERT-base-uncased model as thethe condition of weak supervision, and the performance isbase neural language model. For the three datasetscomparable to the results of supervised classification

AGNews, IMDB and DBpedia, the learning rates are set as

method.

5e-3, 5e-3, 6e-4, and the length of soft prompt is set to 5
for all three datasets. With α set to 20, we repeat the self-

5. Conclusion

optimization process for five times, each containing 5-10

In this paper, we proposed a method using both hard andepochs. The training batch size is 12, and we test all results

soft prompts to solve the text classification problem underon given training dataset. The model is run on one NVIDIA

the situation that only unlabeled documents and label

GeForce GTX 3090 GPU.

names are available. Since the optimization of continuous
prompts requires certain amount of labeled data, while

4.4. Result Comparison and Analysis

manual discrete prompts suffer from instability, we choose

Table 2 presents the performance of our method

to combine these two methods to improve the performance .

compared with other weakly supervised text classification

Firstly, we construct templates and verbalizer, and obtainbaselines and the supervised SOTA method over four

the class distribution of each document by masked

different datasets. Our proposed hard and soft prompt -language model. Then a set of pseudo label prediction

tuning (HSPT) method outperforms previous weakly

results with high confidence are selected as the subsequent

supervised classification methods by about 3 percent points

training data. Then the plausible samples are used toof the Micro-F1 and Macro-F1 scores on 20News, AGNews

optimize the continuous prompt vector added before theinput of every layer in the model. After that, the optimized

model is used to make predictions on the who le unlabeled
data, and part of the prediction results are used to reoptimize the prefix vector. Finally, we can obtain the finalclassification model after repeating this process several
times.
In the experiments, we find that our HSPT method
exhibits competitive performance on four different datasets.
The results on three datasets are outperforming the baselinemodels, while the results on the DBPedia dataset is fallingbehind of X-Class, which may be related to the nature of
the dataset and the construction of the verbalizer.
For future work, we should first continue to improve thehyperparameters of the model and expand the selection ofverbalizers by using other knowledge base. In addition, we
will construct and combine more templates for eachdatasets. Moreover, the soft trainable prompt is added atthe very beginning of the input as a prefix currently,
perhaps investigating the influence of different locationsof prompts on the effect of the model can be taken as thenext research direction, such as adding the continuous
prompt to the end of the input.

References[1] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan,
J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry,G., Askell, A. and Agarwal, S., “Language models are
few-shot learners. Advances in neural information
processing systems”, 33, pp.1877-1901, 2020.
[2] Davison, Joe, Joshua Feldman, and Alexander M.
Rush. “Commonsense knowledge mining frompretrained models.” Proceedings of the 2019conference on empirical methods in natural language
processing and the 9th international joint conference
on natural language processing (EMNLP -IJCNLP), pp.
1173-1178, 2019.
[3] Devlin, Jacob., et al. “BERT: Pre-Training of Deep
Bidirectional
Transformersfor
Language
Understanding.” Proceedings of the 2019 Conf erenceof the North, 2019.
[4] Lehmann, J., Isele, R., Jakob, M., Jentzsch, A.,
Kontokostas, D., Mendes, P.N., Hellmann, S., Morsey,
M., Van Kleef, P., Auer, S. and Bizer, C. “Dbpedia–alarge-scale, multilingual knowledge base extracted
from wikipedia.” Semantic web, 6(2), pp.167-195,
2015.
[5] Lester, Brian, Rami Al-Rfou, and Noah Constant.
“The power of scale for parameter-efficient prompt
tuning.” arXiv preprint arXiv:2104.08691, 2021.
[6] Li, Xiang Lisa, and Percy Liang. “Prefix-tuning:
Optimizing continuous promp ts for generation.”
arXiv preprint arXiv:2101.00190 , 2021.
[7] Liu, Xiao, Yanan Zheng, Zhengxiao Du, Ming Ding,
Yujie Qian, Zhilin Yang, and Jie Tang. “GPT
understands, too.” arXiv preprint arXiv:2103.10385 ,2021.
[8] Liu, Xiao, Kaixuan Ji, Yicheng Fu, Zhengxiao Du,

Zhilin Yang, and Jie Tang. “P-tuning v2: Prompt
tuning can be comparable to fine-tuning universally
acrossscalesandtasks.”
arXiv
preprint
arXiv:2110.07602, 2021.
[9] Liu, Xiao, Kaixuan Ji, Yicheng Fu, Weng Ta m,
Zhengxiao Du, Zhilin Yang, and Jie Tang. “P-Tuning:
Prompt Tuning Can Be Comparable to Fine -tuning
Across Scales and Tasks.” In Proceedings of the 60th
Annual Meeting of the Association for Computational
Linguistics (Volume 2: Short Papers), pp. 61 -68, 2022.
[10] Maas, Andrew, Raymond E. Daly, Peter T. Pham, Dan
Huang, Andrew Y. Ng, and Christopher Potts.
“Learning word vectors for sentiment analysis. ” In
Proceedings of the 49th annual meeting of theassociation for computational linguistics: Human
language technologies, pp. 142-150, 2011.
[11] Meng, Yu, Jiaming Shen, Chao Zhang, and Jiawei Han.
“Weakly-supervised neural text classification. ” Inproceedings of the 27th ACM International
Conferenceon
information
andknowledge
management, pp. 983-992, 2018.
[12] Meng, Yu, Yunyi Zhang, Jiaxin Huang, Chenyan
Xiong, Heng Ji, Chao Zhang, and Jiawei Han. “Textclassification using label names only: A language
model self-training approach.” arXiv preprint
arXiv:2010.07245, 2020.
[13] Pedersen, Ted, Siddharth Patwardhan, and Jason
Michelizzi. “WordNet:: Similarity-Measuring theRelatedness of Concepts.” In AAAI, vol. 4, pp. 25-29,
2004.
[14] Petroni, Fabio, et al. “Language models as knowledge
bases?.” arXiv preprint arXiv:1909.01066, 2019.
[15] Schick, Timo, and Hinrich Schütze. “Exploiting cloze
questions for few shot text classification and natural
languageinference.”
arXiv
preprint
arXiv:2001.07676, 2020.
[16] Schick, Timo, and Hinrich Schütze. “It's not just sizethat matters: Small language models are also few-shotlearners.” arXiv preprint arXiv:2009.07118, 2020.
[17] Shin, Taylor, Yasaman Razeghi, Robert L. Logan IV,
Eric Wallace, and Sameer Singh. “Autoprompt: IV,
Eric Wallace, and Sameer Singh. “Autoprompt:
Eliciting knowledge from language models with
automatically generated prompts. ” arXiv preprint
arXiv:2010.15980, 2020.
[18] Wang, Zihan, Dheeraj Mekala, and Jingbo Shang. “Xclass: Text classification with extremely weaksupervision.” arXiv preprint arXiv:2010.12794, 2020.
[19] Yin, Wenpeng, Jamaal Hay, and Dan Roth.
“Benchmarking zero-shot text classification: Datasets,
evaluation and entailment approach. ” arXiv preprint
arXiv:1909.00161, 2019.
[20] Zhang, Xiang, Junbo Zhao, and Yann LeCun.
“Character-level convolutional networks for textclassification.” Advances in neural information
processing systems 28, 2015.