Exploring Lottery Prompts for Pre-trained Language Models
Year: 2023

Abstract:
Consistently scaling pre-trained language models (PLMs) imposes substantial burdens on model adaptation, necessitating more efficient alternatives to conventional fine-tuning. Given the advantage of prompting in the zero-shot setting and the observed performance fluctuation among different prompts, we explore the instance-level prompt and their generalizability. By searching through the prompt space, we first validate the assumption that for every instance, there is almost always a lottery prompt that induces the correct prediction from the PLM, and such prompt can be obtained at a low cost thanks to the inherent ability of PLMs. Meanwhile, we find that some strong lottery prompts have high performance over the whole training set, and they are equipped with distinguishable linguistic features. Lastly, we attempt to generalize the searched strong lottery prompts to unseen data with prompt ensembling method without any parameter tuning. Experiments are conducted on various types of NLP classification tasks and demonstrate that the proposed method can achieve comparable results with other gradient-free and optimization-free baselines.

heading: 1 Introduction
text: Since pre-trained language models (PLMs) became the de-facto standard in modern NLP researches (Devlin et al., 2019; Liu et al., 2019; Han et al., 2021a), the pretraining-finetuning paradigm has been prevailing until recent years when models keep scaling (Radford et al., 2019; Brown et al., 2020; Rae et al., 2021) and become too expensive to be optimized. To this end, researchers are actively seeking more effective strategies that require little or even no optimization to harness PLMs.
Among these exploratory studies of advanced model adaptation, prompting (Brown et al.,
∗ equal contributions † corresponding authors
2020; Schick et al., 2020; Schick and Schütze, 2021a; Gao et al., 2021) is gaining popularity in the community, which uses additional context (prompts) to wrap input instances and trigger desired output tokens. Note that in this paper, the term “prompt” technically refers to the template that wraps the original input. In classification tasks, these tokens are further mapped to particular labels by a verbalizer. Such a paradigm is verified to be effective in a variety of downstream tasks, even when annotations are insufficient. Particularly, empirical evidence shows that coincidental prompts could achieve extraordinary performance in the zero-shot setting, i.e., no training examples are presented. For example, simple manual prompt can achieve an F1 score of over 60% on 46-class entity typing dataset (Ding et al., 2021a) and reaches 73% accuracy on DBpedia with 14 classes (Hu et al., 2021) in the zero-shot setting.
Despite the promising performance of prompting, it is often accompanied by drastic fluctuations among different prompts (Zhao et al., 2021). Given the observed sensitivity and context-dependent nature of the prompting method, it is intuitive to assign distinct prompts to each instance to trigger the desired output. Intrigued by this intuition, we explore a bold hypothesis:

heading: Is it possible to find at least one instance-
text: level prompt that induces correct output for every data point (lottery prompt) in classification tasks without any optimization?
We empirically show that after building an automatic searching procedure with reasonable searching space on 13 representative classification datasets of up to 66 classes, the existence of such lottery prompts can be validated (§ 2) . That is, the combination of just a few discrete tokens can make a PLM output correct labels for almost any classification data. This finding updates our
15428
recognition of the limit of prompted knowledge in PLMs and demonstrates a promising upper bound of the PLMs’ inference capability.
With the hypothesis verified, we conduct further analysis on the internal mechanisms and properties of lottery prompts to explore how the lottery prompts relate to model capability and how lottery prompts generalize to unseen data without any optimization.
(1) We first find that the search cost of lottery prompts is low for most datasets (under 30 API calls), and could reflect task difficulty and model capacity (§ 3.1). Search success rate increases and search cost decreases for larger PLMs and PLMs pre-trained for more steps, demonstrating that lottery prompts are a unique consequence of the expanded model capacity, rather than a mere stroke of luck. (2) Among these lottery prompts, we also find that there are a number of “strong prompts” that perform non-trivially on the whole training set, and interpretable linguistic features can be identified among them (§ 3.2). Strong prompts demonstrate considerable potential to be generalized to unseen data, i.e., test dataset, of the current task. We develop a mutual-information-based prompt ensembling method and show that strong prompts could be effectively generalized to unseen data in an optimization-free manner (§ 4). Without any parameter update, the ensembling of strong prompts could achieve on-par or better performance with many competitive baselines.
In summary, we validate the existence of lottery prompts and conduct an in-depth analysis of the properties of lottery prompts. We also show that by directly ensembling the strong prompts, prominent performance can be achieved on test data without any optimization. Our study points to the great potential of PLMs and is hoped to inspire future works in more efficient ways in searching and ensembling lottery prompts as an optimization-free adaptation of PLMs.

heading: 2 The Existence of Lottery Prompts for Every Data Point
text: Considering the extraordinary performance observed on zero-shot classification and the large variance brought by the prompt selection, we make an assumption as follows: Given a pre-trained language model and a classification dataset, for each instance, at least one lottery prompt exists that can induce the desired label from the PLM, without the
need to update the PLM parameters.
To validate the assumption, we conduct experiments that attempt to find the lottery prompt for every data point on 13 classification tasks. Note that for different instances, the prompt may be different, and our goal is to verify the existence of such prompts in this experiment.

heading: 2.1 Overview and Setup
text: Particularly, for every input instance in a classification task, we attempt to search through the prompt space and find a textual prompt that can make PLMs produce desired label words. We choose 13 datasets of various NLP tasks for assumption validation. Most of them come from GLUE benchmark (Wang et al., 2018), and others include Yelp Polarity (Zhang et al., 2015), SNLI (Bowman et al., 2015), AG’s News (Zhang et al., 2015), DBpedia (Zhang et al., 2015), and Few-NERD (Ding et al., 2021b). SST-2 (Socher et al., 2013) and Yelp Polarity are datasets for binary sentiment classification. CoLA (Warstadt et al., 2019) is for acceptibility judgment of single sentence. SNLI, RTE (Wang et al., 2018), QNLI (Wang et al., 2018), WNLI (Levesque, 2011) and MNLI (Williams et al., 2018) target at language inference detection given a sentence pair. QQP (Iyer et al., 2017) and MRPC (Schick et al., 2020) are for paraphrase judgment. AG’s News and DBpedia are used for text theme classification. Few-NERD is an entity typing dataset.
As for prompt search space, 200 words with top frequency in English1 are gathered and grouped according to part-of-speech tag with NLTK package (Loper and Bird, 2002) into nouns, verbs, prepositions, adjectives and adverbs. The designed prompt search space is the Cartesian product of three word sets T = NOUNS×VERBS× (PREP∪ ADJ∪ADV)×{<MASK>}, and |T | = 76725. The major concerns of such designing is to restrict the prompt space and to fit with common syntactic order of words to ensure prompt plausibility to some extent. As for verbalizers, we follow the standard design of previous works (Sun et al., 2022). We use RoBERTa-large (Liu et al., 2019) and GPT-2 (Radford et al., 2019) as the backbones. The specific prompt format and verbalizers used are shown in Appendix C.
1https://sketchengine.co.uk

heading: 2.2 The Searching Process
text: For each dataset, we randomly sample 1000 instances from the training set as Xtrain = {(xi, yi)} and apply each prompt T ∈ T to each instance and use the PLM M to produce the prediction. Specifically, a prompt T composed of a noun, a verb and an adjective may be “it was really”. Applying it to an instance x:“A fun movie.” will yeild the input text T (x):“A fun movie. it was really <MASK>”. For each of such pair T (x) ∈ Xtrain × T , the score for each class can be obtained as
o(x;T,M) = Softmax(V(M(T (x)))), (1)
where V denotes the projection from output logits over PLM vocabulary to the class label set. Specifically, to reduce the impact from the prompt, we use calibration (Zhao et al., 2021) to rescale the scores before making the final prediction.
q(T ;M) = Softmax(V(M(T (·)))),
p(x;T,M) = Normalize(o(x;T,M) q(T ;M) ). (2)
T (·) means a wrapped input with empty string and q is the corresponding output probability over the label words. p is the final calibrated probability over the class labels. For every (x, y) ∈ Xtrain, we enumerate over each T ∈ T and see if the output ŷ = argmax p will give the correct prediction y.

heading: 2.3 Verification of the Assumption
text: Table 1 reports the basic searching results. Each instance x is considered correctly predicted if there exists T ∈ T such that y = argmax p. It is shown that for all datasets, a lottery prompt that induces the correct prediction from M exists for almost all 1000 instances. The assumption is thus validated, that is, in a finite search space composed of textual tokens, we can almost always find at least one combination of common words as a prompt to make the prediction correct. While it may not be surprising to see a success on binary classification tasks, achieving 100% coverage on Few-NERD, a 66-class dataset for entity typing, is worth noting. It indicates that the particular semantics distributed in PLM can be triggered by certain contexts even without any further fine-tuning.
Naturally, the phenomenon is not observed when the model is not pre-trained. We conduct the same searching process for Few-NERD on a randomly initialized RoBERTa-large, and only
33.1% instances could find the corresponding lottery prompts. The effect of pre-training will be further explored in Section 3.1, demonstrating that lottery prompts are a unique and consequent effect along with language model pre-training.

heading: 3 Empirical Analysis
text: Since we have verified the existence lottery prompts, in this section, we conduct further analysis on search cost and the searched lottery prompts.

heading: 3.1 Search Cost Analysis
text: As aforementioned, the searching space in our experiment is |T | = 76725, however, the practical cost to find a lottery prompt for one data point is significantly lower than the budget. As shown in Figure 2, the average search cost for each instance does not exceed 30 API calls on most datasets for both PLMs. In this section, we show that search cost correlates with data difficulty and model capacity with further analysis. Task Difficulty. As shown in Figure 2, searching for a lottery prompt for a multi-class classification problem is more costly. The 66-class typing dataset Few-NERD requires a significantly higher search budget than the rest of the datasets, most of which only contain 2 or 3 classes. Another reasonable observation is that single sentence classification tasks are generally easier than tasks involving sentence pairs. As mentioned in the next part, it may be attributed to the designing of prompt format and label words. Meanwhile, NLI tasks with mixed domains are probably the most difficult
500 1000 1500
96
98
100 Agnews
500 1000 1500
96
98
100 FewNERD
2
4
6
500
1000
1500
#Model Parameters (M)
Su cc
es s r
at e
0 5e4 1e599.0
99.5
100.0 Agnews
0 5e4 1e5 85
90
95
100 FewNERD
0
50
100
150
5e3
1e4
1.5e4
Pre-training steps
Se ar
ch c
os t
Figure 1: The change of search success rate and the mean search cost along with the model size (left) and the pre-training steps (right). Experiments are conducted with GPT-2, GPT-2-medium, GPT-2-large, GPT-2-xl, and RoBERTa-base pre-trained with 5000, 10000, 50000, and 100000 steps.
sentence-pair tasks, given that MNLI, RTE, and SNLI are more costly than paraphrase tasks and other domain-specific NLI datasets. Comparing across models, the auto-regressive model (GPT2) generally takes more searches than the autoencoding model (RoBERTa-large). Despite the differences in individual datasets, they show similar trends, which can roughly reflect how difficult the dataset is for PLMs. Hard Instances. Beyond task difficulty, we are also interested in some of the hard instances, i.e. instances that require a significant number of searches or fail to match any lottery prompt in the given search space. We gather the 5 instances that require the most searches or ultimately observe a failure in searching from both PLMs. The examples from 3 datasets are presented in Appendix Table 8. It can be seen that for SST-2, the presented
cases are intuitively difficult, as many of them involve vague expressions and complex reasoning that can be misleading. On the other hand, the hard cases in MNLI and SNLI seem more counterintuitive. Most “entailment” cases have considerable vocabulary overlap between the premise and hypothesis statements. The three failed cases are short sentences with almost identical expressions. We believe it is the negative effect from prompt template and label word chosen. For MNLI, both the high-lighted cases contain negation auxiliaries that rarely follow a “Yes” statement. This tendency drives the PLMs to always favor the choice of “No”, which leads to erroneous prediction. The effect of negation has also been studied with standard PLM finetuning and proved to be a challenge (Hossain et al., 2022; Hosseini et al., 2021). The analysis shows that although for most instances, the lottery prompts can be easily found, the prompting method is still disadvantaged when it comes to complex text that requires advanced understanding ability. Also, prompting method is sensitive to verbalizer designs and can be easily influenced by statistical correlation between label words and input texts. Impact of Model Size and Pre-training. To explore the effect of model capacity on the easiness to search for lottery prompts, we conduct the same searching process as described in § 2 on AG’s News and FewNERD with PLMs of different scales and pre-training status. Specifically, we use GPT2, GPT-2-medium, GPT-2-large and GPT-2-xl for model size ablation and RoBERTa-base pre-trained for 5000∼100000 steps for pre-training ablation, respectively. Figure 1 shows the variation of search success rate and average search cost per instance. For models of different scale, the success rate is similar but the search cost consistently decreases as models scale up, which shows that large PLMs generally have a larger feasible solution space for spe-
cific instances. Meanwhile, finding lottery prompts for PLM at their early pre-training stage is much harder. As the pre-training progresses, a significant reduction in search cost and increase in success rate follow. This indicates that the existence of lottery prompts is not merely a stroke of luck, but a consequence of pre-training that expands model capacity and can be further strengthened as PLMs scale up.

heading: 3.2 The Strong Prompts
text: After searching for lottery prompts for all instances, we are interested in if there are “strong prompts” among them, i.e., prompts that perform well on the whole Xtrain. We measure the performance of each prompt over Xtrain with standard metrics on some representative datasets from each task category. The metric statistics and variation of all prompts are shown in Figure 3. It could be concluded that for all datasets, there are a handful of “strong prompts” that can perform satisfactorily on the dataset. Note that despite having altogether 66 classes, the best-performing prompt almost achieves an accuracy of 0.4 on Few-NERD. Meanwhile, different tasks show distinct patterns. Text classification tasks with single sentence are more sensitive to prompt choice and often observe larger performance variation over the prompt space. For SST-2, while the best-performing prompt reaches an accuracy of 0.8, the worst prompts can barely get to 0.3. For NLI tasks, prompt performance is more stable however mediocre.
To inspect into the linguistic characteristics of the strong prompts, we present the top-5 prompts for some of the representative datasets and their
corresponding metrics on the training set of 1000 instances in Table 2. While many prompts may not seem syntactical on the whole, certain linguistic characteristics can still be identified, which fit with our language intuition, both syntactically and semantically, and reveal some of the most contributive words in prompts for distinctive datasets. For example, the top prompts for the sentiment analysis task are compatible with chosen label words. Adverbs that enhance the statement (e.g. just, really, very) appear frequently in sentiment analysis tasks. For topic classification, the words like “other” and “such” naturally lead to nominal label words like “sports” and “artist”. As for natural language inference task, although language entailment is subjective, it is common that personal pronouns are often involved when we express our opinions on entailment, like "I think it means", "Do you think", etc. Therefore appearance pf pronouns is in top prompts is reasonable. Meanwhile, we do observe that good prompts are not always interpretable. It may imply that the PLM’s internal language ability and understanding deviates from human beings, which is why prompt engineering is important. Above all, we see that “strong prompts” do exist and they are equipped with distinct linguistic features depending on label words and task type.

heading: 4 Explore the Generalizability of Strong Prompts
text: In § 2 and § 3, we have empirically verified that conditioned on a pre-trained model and a classification task, it is possible to find a lottery prompt for almost every data point, and that there are a handful of strong prompts that perform non-trivially on the training set. In this section, we first describe the ensembling method and then present the generalization results.

heading: 4.1 Prompt Ensembling Method
text: We gather a set of feasible prompts T ∗ with the searching result on Xtrain and conduct inference with designed prompt ensembling method for each instance in Xtest. Since the choice of T ∗ is solely based on inference results on Xtrain, the process uses no validation set. Formally, given the selected prompts T ∗ = {T1, T2, ..., Tt} ⊂ T , the prediction for each data point x ∈ Xtest is presented as p(x; T ∗,M) = Φ(p1, p2, ..., pt), (3) where pk = p(x;Tk,M) and is calculated as equation 2, and Φ is the ensembling method used.
Dataset Top-5 Prompts Metrics
SST-2 he work just, I find very, I find really, help are for, she work just 85.9, 85.6, 85.2, 84.6, 84.0 Yelp P. look place really, you place also, look was also, I were very, they place also 92.0, 91.3, 91.3, 91.2, 91.2
SNLI I get really, I like through, I said always, keep love through, you found that 56.9, 56.0, 55.8, 55.8, 55.7 RTE keep like always, way think such, life think same, end think such, end like always 60.0, 59.7, 59.6, 59.6, 59.4
MRPC money had very, something had very, I been very, help had very, life had very 70.9, 70.5, 70.4, 70.4, 70.2
AG’s News lot say on, I said other, time think other, state say on, you think other 79.7, 78.8, 78.1, 78.0, 77.3 DBpedia you said such, something know then, life make of, home said such, information is that 87.5, 86.6, 86.0, 85.9, 85.8
Table 2: An example of Top-5 prompts over 1000 training instances for each dataset and their individual performance on training sets. The model used is RoBERTa-large.
Textual Input
Too bad. The film is strictly routine. A fun ride. …
I, he, company, …
make, find, love, …
really, good, even …
[MASK]
[NOUN] [VERB] [PREP|ADJ|ADV] [MASK]
× × × × =
Searching Space
Searching Positions
Searched Prompts
Too bad. I make really [MASK] Too bad. I make good [MASK] Too bad. He finds very [MASK] A fun ride. I make even [MASK] A fun ride. I find really [MASK] A fun ride. I find good [MASK] …
I make really [MASK] I make good [MASK] I make even [MASK] I find really [MASK] I find good [MASK] …
Lottery Prompts
Too bad. The film is strictly routine. A fun ride. …
Input
×
Frozen Pre-trained Language Model (Only Inference Available)
I make really [MASK] I make good [MASK] I make even [MASK] I find really [MASK] …
Strong PromptsTop K
A cc
. o n
Prompt ID 1 2…K
Test Input
Really boring. It has bright scenes. What a fantastic show. … ×
En se
m bl
in g
Simple Voting
Mutual Information
Really boring.
T1
<latexit sha1_base64="VE7EO5qDB9PpT2xmOiA3YBELutc=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LFbBU0mkoseCF48VW1toQ9lsN+3SzSbsToQS+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IJHCoOt+O4W19Y3NreJ2aWd3b/+gfHj0aOJUM95isYx1J6CGS6F4CwVK3kk0p1EgeTsY38789hPXRsSqiZOE+xEdKhEKRtFKD82+1y9X3Ko7B1klXk4qkKPRL3/1BjFLI66QSWpM13MT9DOqUTDJp6VeanhC2ZgOeddSRSNu/Gx+6pScW2VAwljbUkjm6u+JjEbGTKLAdkYUR2bZm4n/ed0Uwxs/EypJkSu2WBSmkmBMZn+TgdCcoZxYQpkW9lbCRlRThjadkg3BW355lTxeVr1a9eq+Vqmf5XEU4QRO4QI8uIY63EEDWsBgCM/wCm+OdF6cd+dj0Vpw8plj+APn8wfOW41m</latexit>
T2
<latexit sha1_base64="yXyT97HHIATiNry8UgLe4M1X2vw=">AAAB6nicbVDLSgNBEOyNrxhfUY9eBqPgKeyGiB4DXjxGzAuSEGYnvcmQ2dllZlYISz7BiwdFvPpF3vwbJ8keNLGgoajqprvLjwXXxnW/ndzG5tb2Tn63sLd/cHhUPD5p6ShRDJssEpHq+FSj4BKbhhuBnVghDX2BbX9yN/fbT6g0j2TDTGPsh3QkecAZNVZ6bAwqg2LJLbsLkHXiZaQEGeqD4ldvGLEkRGmYoFp3PTc2/ZQqw5nAWaGXaIwpm9ARdi2VNETdTxenzsilVYYkiJQtachC/T2R0lDraejbzpCasV715uJ/XjcxwW0/5TJODEq2XBQkgpiIzP8mQ66QGTG1hDLF7a2EjamizNh0CjYEb/XlddKqlL1q+fqhWqpdZHHk4QzO4Qo8uIEa3EMdmsBgBM/wCm+OcF6cd+dj2ZpzsplT+APn8wfP341n</latexit>
√
NEGATIVE
3
POLL
POSITIVE
T3
<latexit sha1_base64="GnC8FXWfbT+mrFC7fAcpfk3C5Wo=">AAAB6nicbVDLSgNBEOyNrxhfUY9eBqPgKexqRI8BLx4j5gXJEmYnvcmQ2dllZlYIIZ/gxYMiXv0ib/6Nk2QPmljQUFR1090VJIJr47rfTm5tfWNzK79d2Nnd2z8oHh41dZwqhg0Wi1i1A6pRcIkNw43AdqKQRoHAVjC6m/mtJ1Sax7Juxgn6ER1IHnJGjZUe672rXrHklt05yCrxMlKCDLVe8avbj1kaoTRMUK07npsYf0KV4UzgtNBNNSaUjegAO5ZKGqH2J/NTp+TcKn0SxsqWNGSu/p6Y0EjrcRTYzoiaoV72ZuJ/Xic14a0/4TJJDUq2WBSmgpiYzP4mfa6QGTG2hDLF7a2EDamizNh0CjYEb/nlVdK8LHuV8vVDpVQ9y+LIwwmcwgV4cANVuIcaNIDBAJ7hFd4c4bw4787HojXnZDPH8AfO5w/RY41o</latexit>
T4
<latexit sha1_base64="GXa1Dh9zZWjbN69tqOUkzaHE/Io=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LFbBU0mkoseCF48VW1toQ9lsJ+3SzSbsboQS+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IBFcG9f9dgpr6xubW8Xt0s7u3v5B+fDoUcepYthisYhVJ6AaBZfYMtwI7CQKaRQIbAfj25nffkKleSybZpKgH9Gh5CFn1Fjpodmv9csVt+rOQVaJl5MK5Gj0y1+9QczSCKVhgmrd9dzE+BlVhjOB01Iv1ZhQNqZD7FoqaYTaz+anTsm5VQYkjJUtachc/T2R0UjrSRTYzoiakV72ZuJ/Xjc14Y2fcZmkBiVbLApTQUxMZn+TAVfIjJhYQpni9lbCRlRRZmw6JRuCt/zyKnm8rHq16tV9rVI/y+MowgmcwgV4cA11uIMGtIDBEJ7hFd4c4bw4787HorXg5DPH8AfO5w/S541p</latexit> 0.38
Prob. p
<latexit sha1_base64="wSPewWmd8uqyCsd1to8eznSpOKE=">AAAB6HicbVBNS8NAEJ34WetX1aOXxSp4KolU9Fjw4rEF+wFtKJvtpF272YTdjVBCf4EXD4p49Sd589+4bXPQ1gcDj/dmmJkXJIJr47rfztr6xubWdmGnuLu3f3BYOjpu6ThVDJssFrHqBFSj4BKbhhuBnUQhjQKB7WB8N/PbT6g0j+WDmSToR3QoecgZNVZqJP1S2a24c5BV4uWkDDnq/dJXbxCzNEJpmKBadz03MX5GleFM4LTYSzUmlI3pELuWShqh9rP5oVNyYZUBCWNlSxoyV39PZDTSehIFtjOiZqSXvZn4n9dNTXjrZ1wmqUHJFovCVBATk9nXZMAVMiMmllCmuL2VsBFVlBmbTdGG4C2/vEpaVxWvWrluVMu18zyOApzCGVyCBzdQg3uoQxMYIDzDK7w5j86L8+58LFrXnHzmBP7A+fwB06uM3g==</latexit>
A cc
. o n
Prompt ID
Mutual Information
T1
<latexit sha1_base64="VE7EO5qDB9PpT2xmOiA3YBELutc=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LFbBU0mkoseCF48VW1toQ9lsN+3SzSbsToQS+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IJHCoOt+O4W19Y3NreJ2aWd3b/+gfHj0aOJUM95isYx1J6CGS6F4CwVK3kk0p1EgeTsY38789hPXRsSqiZOE+xEdKhEKRtFKD82+1y9X3Ko7B1klXk4qkKPRL3/1BjFLI66QSWpM13MT9DOqUTDJp6VeanhC2ZgOeddSRSNu/Gx+6pScW2VAwljbUkjm6u+JjEbGTKLAdkYUR2bZm4n/ed0Uwxs/EypJkSu2WBSmkmBMZn+TgdCcoZxYQpkW9lbCRlRThjadkg3BW355lTxeVr1a9eq+Vqmf5XEU4QRO4QI8uIY63EEDWsBgCM/wCm+OdF6cd+dj0Vpw8plj+APn8wfOW41m</latexit>
T2
<latexit sha1_base64="yXyT97HHIATiNry8UgLe4M1X2vw=">AAAB6nicbVDLSgNBEOyNrxhfUY9eBqPgKeyGiB4DXjxGzAuSEGYnvcmQ2dllZlYISz7BiwdFvPpF3vwbJ8keNLGgoajqprvLjwXXxnW/ndzG5tb2Tn63sLd/cHhUPD5p6ShRDJssEpHq+FSj4BKbhhuBnVghDX2BbX9yN/fbT6g0j2TDTGPsh3QkecAZNVZ6bAwqg2LJLbsLkHXiZaQEGeqD4ldvGLEkRGmYoFp3PTc2/ZQqw5nAWaGXaIwpm9ARdi2VNETdTxenzsilVYYkiJQtachC/T2R0lDraejbzpCasV715uJ/XjcxwW0/5TJODEq2XBQkgpiIzP8mQ66QGTG1hDLF7a2EjamizNh0CjYEb/XlddKqlL1q+fqhWqpdZHHk4QzO4Qo8uIEa3EMdmsBgBM/wCm+OcF6cd+dj2ZpzsplT+APn8wfP341n</latexit>
T3
<latexit sha1_base64="GnC8FXWfbT+mrFC7fAcpfk3C5Wo=">AAAB6nicbVDLSgNBEOyNrxhfUY9eBqPgKexqRI8BLx4j5gXJEmYnvcmQ2dllZlYIIZ/gxYMiXv0ib/6Nk2QPmljQUFR1090VJIJr47rfTm5tfWNzK79d2Nnd2z8oHh41dZwqhg0Wi1i1A6pRcIkNw43AdqKQRoHAVjC6m/mtJ1Sax7Juxgn6ER1IHnJGjZUe672rXrHklt05yCrxMlKCDLVe8avbj1kaoTRMUK07npsYf0KV4UzgtNBNNSaUjegAO5ZKGqH2J/NTp+TcKn0SxsqWNGSu/p6Y0EjrcRTYzoiaoV72ZuJ/Xic14a0/4TJJDUq2WBSmgpiYzP4mfa6QGTG2hDLF7a2EDamizNh0CjYEb/nlVdK8LHuV8vVDpVQ9y+LIwwmcwgV4cANVuIcaNIDBAJ7hFd4c4bw4787HojXnZDPH8AfO5w/RY41o</latexit>
T4
<latexit sha1_base64="GXa1Dh9zZWjbN69tqOUkzaHE/Io=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LFbBU0mkoseCF48VW1toQ9lsJ+3SzSbsboQS+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IBFcG9f9dgpr6xubW8Xt0s7u3v5B+fDoUcepYthisYhVJ6AaBZfYMtwI7CQKaRQIbAfj25nffkKleSybZpKgH9Gh5CFn1Fjpodmv9csVt+rOQVaJl5MK5Gj0y1+9QczSCKVhgmrd9dzE+BlVhjOB01Iv1ZhQNqZD7FoqaYTaz+anTsm5VQYkjJUtachc/T2R0UjrSRTYzoiakV72ZuJ/Xjc14Y2fcZmkBiVbLApTQUxMZn+TAVfIjJhYQpni9lbCRlRRZmw6JRuCt/zyKnm8rHq16tV9rVI/y+MowgmcwgV4cA11uIMGtIDBEJ7hFd4c4bw4787HorXg5DPH8AfO5w/S541p</latexit>
q log q
<latexit sha1_base64="NFws8lAXGKZyClTOl/wCpZRyVyY=">AAAB73icbVBNSwMxEJ2tX7V+VT16CVbBi2VXKnosePFYwX5Au5Rsmm1Ds8k2yQpl6Z/w4kERr/4db/4bs+0etPXBwOO9GWbmBTFn2rjut1NYW9/Y3Cpul3Z29/YPyodHLS0TRWiTSC5VJ8CaciZo0zDDaSdWFEcBp+1gfJf57SeqNJPi0Uxj6kd4KFjICDZW6lxOelwO0aRfrrhVdw60SrycVCBHo1/+6g0kSSIqDOFY667nxsZPsTKMcDor9RJNY0zGeEi7lgocUe2n83tn6NwqAxRKZUsYNFd/T6Q40noaBbYzwmakl71M/M/rJia89VMm4sRQQRaLwoQjI1H2PBowRYnhU0swUczeisgIK0yMjahkQ/CWX14lrauqV6teP9Qq9bM8jiKcwClcgAc3UId7aEATCHB4hld4cybOi/PufCxaC04+cwx/4Hz+AHToj4E=</latexit>
p log p
<latexit sha1_base64="AKL2rUMIRXISg/frwuto/GyWiww=">AAAB7nicbVBNSwMxEJ2tX7V+VT16CVbBU9mVFj0WvHisYD+gXUo2zbah2SQkWaEs/RFePCji1d/jzX9j2u5BWx8MPN6bYWZepDgz1ve/vcLG5tb2TnG3tLd/cHhUPj5pG5lqQltEcqm7ETaUM0FblllOu0pTnEScdqLJ3dzvPFFtmBSPdqpomOCRYDEj2Dqpo/pcjpAalCt+1V8ArZMgJxXI0RyUv/pDSdKECks4NqYX+MqGGdaWEU5npX5qqMJkgke056jACTVhtjh3hi6dMkSx1K6ERQv190SGE2OmSeQ6E2zHZtWbi/95vdTGt2HGhEotFWS5KE45shLNf0dDpimxfOoIJpq5WxEZY42JdQmVXAjB6svrpH1dDWrV+kOt0rjI4yjCGZzDFQRwAw24hya0gMAEnuEV3jzlvXjv3seyteDlM6fwB97nDwd7j0g=</latexit>
0.62
Inference Results
√ √ × √ √ √ × ×
Figure 4: The complete process of searching for T ∗ and ensembling with Φ.
With the assumption that strong prompts over Xtrain are also expected to perform well on Xtest, these best-performing prompts are regarded as the most reliable for predicting the unseen data. So we take the top-k best-performing prompts over the training set as T ∗. In the experiments, we empirically use k = 10. A naive ensembling method is to take the average output as the final prediction by simple voting, where Φ(p1, p2, ..., pt) = 1t ∑t k=1 pk. While a more sophisticated strategy that echoes the spirit of “lottery prompt” is to select one most “reliable” prompt for each instance x ∈ Xtest. Intuitively, the more reliable a prompt T is, the more confident the model M will be about instance x. Inspired by Sorensen et al. (2022), we measure the confidence with the mutual information between x and y, T , which is defined by the reduction in entropy of predicted probability brought by x,
I(x;Tk,M) = H(q|Tk(·))−H(p|Tk(x)) =− ∑
i qi(Tk;M) log qi(Tk;M)+ ∑
i
pi(x;Tk,M) log pi(x;Tk,M), (4)
where q and p are the predicted probability vectors as in Equation 2. So the overall objective is
T ∗ = arg max T∈T ∗ I(x;T,M),
Φ(p1,p2, ..., pt) = p(x;T ∗,M).
(5)
Specifically, maximization of mutual information entails that a good prompt itself should contain no bias towards the label set, so q should be close to a uniform distribution. On the other hand, a suitable prompt for a specific instance should induce an almost certain prediction on the desired class, corresponding to a near one-hot vector p. Experiments show that under few-shot settings, our mutual-information-based ensembling strategy is more advantageous than direct simple voting (§ 4.2). The complete searching and ensembling process is shown in Figure 4.

heading: 4.2 In-Dataset Generalization
text: Experimental Setup. We comprehensively evaluate the generalizability of strong prompts on 8 representative datasets. Following previous works (Sun et al., 2022), we conduct experiments under few-shot settings. Specifically, we choose
the top-10 prompts as T ∗ and obtain the final prediction and test metrics with mutual-informationbased ensembling as Φ on the test set. We keep the verbalizers aligned with Sun et al. (2022) for fair comparison. The description of experimental details and baselines can be found in Appendix A.
Overall Results. Figure 5 shows the in-dataset generalization results on each dataset. Overall, our method performs comparably to the existing baselines and requires the fewest trainable parameters. For some datasets, the searched strong prompts are shown to be more effective than baselines. It points to the fact that with a reasonable prompt search space and a few training instances, strong prompts can be identified and generalized effectively to unseen data. Best prompt on 32-shot data surprisingly overtakes many baselines. This, jointly with the mediocre performance of manual prompts, indicate that a human-comprehensible prompt may not always be the best choice for PLMs and may fail to probe a considerable amount of intrinsic knowledge in PLMs. Meanwhile, the success of MI over best prompt shows that ensembling a set of strong prompts is beneficial. Comparing across datasets, our method is more advantageous for harder tasks, including natural language inference (SNLI and RTE) and paraphrasing (MRPC). For single-sentence classification tasks, the improvement is minor. This finding fits with our intuition, as tasks involving two sentences often require more
abstract abilities like reasoning and the contexts are more diverse across instances. Designing or optimizing for one unified prompt for such datasets is admittedly harder. Above all, it is exciting that ensembling a proper set of prompts composed of textual tokens may surpass network optimization on a dataset in an era of pre-trained language models and points to the values of mining and tapping into an optimal usage of plain textual prompt.
Impact of Training Data Size. To further explore the property of our method, experiments are conducted under few-shot settings ranging from 8 shots to 256 shots with both simple voting and mutualinformation-based ensembling. As shown in Figure 6, we can see that performance varies a lot when different instances are sampled as the training set under low-shot settings. It suggests the importance of choosing the proper training data for our method. When more shots are provided, metrics get higher and variance gets smaller. As the data volume climbs up to 128 shots and 256 shots, the increase in metrics becomes minor for most datasets. It can also be concluded that for low-shot settings, mutual-information-based ensembling method yields higher results than simple voting. But as more training data are available, the gap is narrowed and the two ensembling strategies converge to similar levels.
Task Setting Metrics
Sentiment Analysis SST-2 → Yelp P. 90.27 ( 1.58 ↓) Yelp P. → SST-2 84.15 ( 5.37 ↓)
Natural Language Inference RTE → SNLI 40.48 ( 10.13 ↓) SNLI → RTE 54.51 (4.19 ↓) MNLI → SNLI 47.96 (2.65 ↓) MNLI → RTE 55.81 (2.89 ↓)
Table 3: Transferability test of T ∗ across datasets with similar tasks. Prompts are searched on 32-shot training data from source dataset and evaluated on test set of target dataset. Top-10 prompts are used as T ∗ and mutual-information-based strategy is used as Φ.

heading: 4.3 Cross-Dataset Generalization
text: We test the prompt transferbility across datasets with similar tasks under 32-shot setting. Experiments are conducted on sentiment analysis and language inference tasks. We also use MNLI as the source dataset as many previous works do. Table 3 shows that prompts chosen by our method can be transferable. While SST-2 and Yelp observe mutual transferability, transfering RTE to SNLI is relatively hard, which can be attributed to the inconsistency in class number. MNLI is shown to be a robust dataset for NLI task and the searched prompts perform satisfactorily on both RTE and SNLI. It is also in line with previous research findings that using prompts pretrained on MNLI could greatly boost performance on other NLI datasets. Above all, the results demonstrate that our proposed strategy can effectively extract representative prompts for a specific kind of task, which can be further utilized to reduce search cost.

heading: 5 Related Work
text: Prompting, as an alternative to standard finetuning, is originally inspired by GPT-3 (Brown et al., 2020) and knowledge probing (Petroni et al.,
2019; Jiang et al., 2020). With a similar form to pre-training tasks, it stimulates the intrinsic knowledge in PLMs more efficiently. Following several of the earliest works (Schick and Schütze, 2021a,b), prompting has been applied in various NLP tasks (Han et al., 2021b; Li and Liang, 2021; Sainz et al., 2021; Ding et al., 2021a). It is also discovered that the specific prompt used has a great impact on task performance. Therefore, efforts have been devoted to prompt engineering and automatic prompt generation. Optimizing for a good prompt has been conducted at both discrete token level (Shin et al., 2020; Gao et al., 2021) and continuous embedding level (Li and Liang, 2021; Zhang et al., 2021; Liu et al., 2021b; Li et al., 2022). Some also focus on the choice and representation of label words (Schick et al., 2020; Hu et al., 2021; Zhang et al., 2021). Experiments show that a well-optimized or pre-trained (Gu et al., 2022) prompt can be beneficial.
Given the striking performance of prompting under few-shot settings especially, recently, more works are focusing on the more efficient tuning of PLMs based on prompts. Prompt tuning (Lester et al., 2021) tunes the pre-pended token embedding only. Other works enhance PLMs’ zero-shot learning ability with prompts. Studies show that large PLMs with proper prompts (Wei et al., 2021) and training with diverse prompts (Sanh et al., 2021) can advance zero-shot performance. This line of work emphasizes the efficient tuning and steering process of large PLMs. Black-box tuning (Sun et al., 2022) optimizes the pre-pended continuous prompt in a projected low-dimensional space without PLM gradient information.
This work is among the first few efforts (Jin et al., 2022; Wu et al., 2022) in mining instancelevel prompts, and is the first to propose and prove the existence of a lottery prompt composed of a few textual tokens for each instance. In contrast to tun-
ing a small number of parameters or tuning without gradients, an optimization-free method is proposed to generalize the searched prompts to the test sets.

heading: 6 Conclusion
text: In this work, we explore the existence of lottery prompts for every single instance and the adaptation of them for various classification tasks in an optimization-free manner. We propose a large prompt space composed of common words as the search space to verify the assumption. We also identify the searched strong prompts and the relation between model capacity and search cost and demonstrate the effectiveness of ensembling the strong prompts on the test set. Our proposed optimizationfree method achieves satisfactory results on various NLP tasks under few-shot settings. Above all, this work illuminates the fact that the great potential of PLMs can be successfully harnessed and prompted by plain textual prompts mined from PLM vocabulary without parameter optimization and thus points to the need for future efforts in more efficient ways in mining and utilizing lottery prompts.

heading: Acknowledgements
text: This research is supported by the National Natural Science Foundation of China (Grant No.62276154 and No.62236004), Research Center for Computer Network (Shenzhen) Ministry of Education, Beijing Academy of Artificial Intelligence (BAAI), the Natural Science Foundation of Guangdong Province (Grant No. 2023A1515012914), Basic Research Fund of Shenzhen City (Grant No. JCYJ20210324120012033 and JSGG20210802154402007), the Major Key Project of PCL for Experiments and Applications (PCL2021A06), Overseas Cooperation Research Fund of Tsinghua Shenzhen International Graduate School (HW2021008), Major Project of the National Social Science Foundation of China (No. 22&ZD298), and Institute Guo Qiang at Tsinghua University. Finally, we thank Xiaozhi for providing valuable comments.
Limitations
The current method works with a large prompt search space T , which means a tremendous number of inference API calls are required. Though Figure 2 shows that the average cost of finding a lottery prompt for each instance is low, the searching
process is highly randomized and there is no guarantee of the performance of searched prompts over the test dataset. Therefore, finding strong prompts over the training set can still be laborious. How to use PLM inference calls more efficiently and leverage the generalization ability of T ∗ within a reasonable cost is of future research interest. Our acceleration strategy can be found in Appendix B.
Another aspect is that not all strong prompts are interpretable as presented in 2. While recently emerged larger models like ChatGPT demonstrate superb language understanding ability and can almost always answer yes or no questions correctly given a human-interpretable prompt. This gap observed between small PLMs like RoBERTa and large language models like ChatGPT is yet another interesting research topic.
Ethical Considerations
This work shows that with proper plain textual prompts, instance-level desired results can be prompted from PLMs. This inherent feature of PLMs means attacks can be launched to produce rude or discriminated words. On the other hand, however, we believe it can also be a technique used for debiasing a PLM. Overall, this effect depends on the intention of the users and the pre-training corpus of the corresponding PLMs. The analysis of this study can be used to facilitate the community to develop more specifications for the rational use of PLMs (especially the super-large ones), and more approaches to effectively prevent potential ethical issues. For example, we can use this technique to analyze which outputs that may have ethical issues are easily triggered by which contexts (prompts) and develop a set of intervention methods to make these tokens unavailable for output.

heading: A Experimental Details
text: A.1 Experimental Settings We conduct experiments under few-shot settings on 8 datasets: SST-2, Yelp P., AG’s News, DBpedia, MRPC, SNLI, RTE and MNLI. We experiment under both 16-shot and 32-shot data as the training set as our method requires no validation set. The total seen labeled data number does not exceed 32- shot across all methods. The original validation set is used as the test set following (Sun et al., 2022). The detailed training and test set statistics for experiments in Figure 5 are shown in Table 4. All datasets are distributed under either CC BY license or CC BY-SA license, or subject to specified term of use. We have read and complied to the terms during experiments. The label words used follow (Sun et al., 2022) and are the same across all methods.
A.2 Baselines We choose comparable baselines that do not update the PLM parameters, including 1) Gradient-based methods: Prompt Tuning and P Tuning v2; 2) Optimization-based methods: Feature MLP, Blackbox Tuning, and RLPrompt; and 3) Optimizationfree methods: Manual Prompt, and In-Context learning. The details are as follows: Prompt Tuning (Lester et al., 2021) optimizes the continuous prompt at the input level. P-Tuning v2 (Liu et al., 2021a) is a variant of prompt tuning that pre-pends trainable parameters to each layer of the PLM and optimizes them in a multi-task setting. FeatureMLP uses pre-trained features output by PLMs and train a lightweight classifier offline. Black-Box Tuning (Sun et al., 2022) is a gradient-free method that optimizes the projected extra 500 parameters at the input layer with Covariance Matrix Adaptation Evolution Strategy. RLPrompt (Deng et al., 2022) optimizes for discrete prompts with reinforcement
learning. Manual Prompt is a zero-shot method that directly uses a hand-crafted textual prompt for each dataset. In-Context Learning (Brown et al., 2020) is an optimization-free method that uses a few samples as demonstrations prepended to the test sample. Table 5 lists the features and trainable parameter number of baselines and our method.
A.3 Implementation Details
RoBERTa-large contains 354 million parameters and GPT-2 has 1.5 billion parameters. There is no extra parameter added in our method. For each dataset, the experiments are run with 5 different random seeds, and the mean metrics are reported. Most baseline results are taken from Sun et al. (2022) and Deng et al. (2022), while we re-run RLPrompt for MRPC and all baselines for MNLI with original code. All experiments are conducted on NVIDIA A100 and GeForce RTX 3090 GPUs with CUDA. The search process in § 4.2 with 32- shot data takes about 2 hours with 40 GB maximum memory. The test process takes 5∼30 minutes depending on the size of T ∗ and Xtest. Our method is developed by OpenPrompt (Ding et al., 2022), an open-source prompt-learning framework based on PyTorch (Paszke et al., 2019). The models are obtained from the Huggingface Transformers library (Wolf et al., 2020).

heading: B Efficiency Analysis
text: The results reported in § 4.2 all search through the whole prompt space T ∗, i.e. every combination of an instance and a prompt is covered. Since it would require up to 4 hours with a single NVIDIA A100, we seek to optimize the process by prun-
ing the search space. Our strategy is as follows: (1) randomly sample a batch of valid prompts (in our experiments we use batch size 16) from T ∗ and apply them to the whole training set Xtrain; (2) record the performance of each prompt word, i.e. if a prompt is “it was really” and achieves 0.8 accuracy on Xtrain, then for each word in the prompt (“it”, “was”, “really”) 0.8 is recorded; (3) update the set of valid prompts; (4) repeat until there is no remaining valid prompt. A valid prompt means the average score of the three words is over a predefined threshold. In our experiments, the threshold is set to 0.7 on SST-2 dataset and achieves a mean test accuracy of 87.6%.
As shown in Table 6, with the pruning strategy, the average time cost can be greatly reduced to 10 minutes with a still satisfying performance on test data. In our experiments, prompts are randomly sampled and grouped into batches. We believe a better-designed and heuristically informed batching strategy will further boost the searching efficiency and test performance.

heading: C Details of Prompts and Label Words
text: Table 7 displays the specific prompt format and label words used for searching for lottery prompt for each dataset with RoBERTa-large. For autoregressive PLMs like GPT-2, the “<mask>” token are removed and the prediction of the next token by PLM will be extracted.
D Visualization of Words in Strong Prompts
We also get the 100 best prompts out of T for SST2, and visualize the frequent words at each position, as shown in Figure 7. From the variation of words
at each position, we can conclude that words more adjacent to the “<mask>” token has a larger impact on the prediction, which fits with our intuition. In addition, GPT-2 demonstrates better fluency and interpretability compared to RoBERTa-large, as some high-frequency words found for RoBERTalarge like “without” are hard to comprehend.
to get the lottery prompt out of T . Instances in purple indicate failure to find a lottery prompt for GPT-2, and instances in blue are failure instances for RoBERTa-large.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
In section "Limitations" at the end of the paper
3 A2. Did you discuss any potential risks of your work? In section "Ethical Considerations" at the end of the paper
3 A3. Do the abstract and introduction summarize the paper’s main claims? See Abstract and Section 1 Introduction
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 2, 3 and 4
3 B1. Did you cite the creators of artifacts you used? Section 2.1
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Appendix A
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix A
C 3 Did you run computational experiments? Left blank.
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Appendix A
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Appendix A
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4.2
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 2.1
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: How to use PLM inference calls more efficiently and leverage the generalization ability of T ∗ within a reasonable cost is of future research interest. Our acceleration strategy can be found in Appendix B. Another aspect is that not all strong prompts are interpretable as presented in 2. While recently emerged larger models like ChatGPT demonstrate superb language understanding ability and can almost always answer yes or no questions correctly given a human-interpretable prompt. This gap observed between small PLMs like RoBERTa and large language models like ChatGPT is yet another interesting research topic. Ethical Considerations
This work shows that with proper plain textual prompts, instance-level desired results can be prompted from PLMs. This inherent feature of PLMs means attacks can be launched to produce rude or discriminated words. On the other hand, however, we believe it can also be a technique used for debiasing a PLM. Overall, this effect depends on the intention of the users and the pre-training corpus of the corresponding PLMs. The analysis of this study can be used to facilitate the community to develop more specifications for the rational use of PLMs (especially the super-large ones), and more approaches to effectively prevent potential ethical issues. For example, we can use this technique to analyze which outputs that may have ethical issues are easily triggered by which contexts (prompts) and develop a set of intervention methods to make these tokens unavailable for output