What Makes Pre-trained Language Models Better Zero-shot Learners?
Year: 2023

Abstract:
and Section

heading: 1 Introduction
text: Prompt learning has been demonstrated to be a successful remedy for challenges associated with pre-training and fine-tuning paradigm, especially in zero/few-shot scenarios (Gao et al., 2021; Schick and Schütze, 2021a,b; Tam et al., 2021; Lu et al., 2022a).
Research has repeatedly shown that various transformer-based language models can benefit from prompt learning. For example, decoder-only models, such as those in the GPT family (Brown et al., 2020), can better generalise to unseen cases by prefixing inputs with a few training examples (in natural language). This is known as in-context learning (Brown et al., 2020; Xie et al., 2021; Liu et al., 2022a). Encoder-decoder models, such as T5 (Raffel et al., 2020) or BART (Lewis et al., 2020), can leverage prompt learning to train versatile models for multiple tasks (Khashabi et al.,
+Work was done during internship at SenseTime Research *Corresponding author
2020; Lester et al., 2021). Bidirectional encoderonly models, such as those in the BERT family (Devlin et al., 2018; Liu et al., 2019), can also manifest impressive zero-shot capacity when given proper prompts. These prompts often take the form of pre-training tasks, such as next sentence prediction (Sun et al., 2022) or masked language modeling (MLM) (Gao et al., 2021; Schick and Schütze, 2021a,b; Tam et al., 2021)—also known as clozestyle prompt learning.
Despite its success in encoder-only models, cloze-style prompt learning is sensitive to the specific involved templates. Multiple studies have shown that the design and choice of prompt templates greatly affect the effectiveness of zero-shot learning (Tam et al., 2021; Zhao et al., 2021; Rubin et al., 2022). Ideally, they are supposed to be as close as possible to the language used in downstream task. For example, in a sentiment analysis task, a suitable template may be “[very/not] pleased.” that carries emotional information. However, other templates can also be used here like “[very/not] good.”. As shown in Table 1, the performance of zero-
shot learning using different sentiment-bearing templates can fluctuate significantly with different prompt templates. For the ECOMMERCE dataset, the template “[very/not] pleased.” achieves the best zero-shot accuracy of 73.12%, while using the template “[very/not] good.” results in an accuracy of only 55.68%—which is only slightly better than random guessing. Additionally, if we choose a sentiment-irrelevant template “[yellow/green] black.”, the accuracy significantly drops to 50.49%, indicating that the model has no classification ability. This shows that the performance of the model is largely shaped by templates used. Therefore, selecting the most appropriate templates for downstream tasks is crucial in zero-shot learning.
Current prompt learning methods still rely on a development set of human-annotated data for
2288
post-hoc template selection (Tam et al., 2021; Sun et al., 2022; Gao et al., 2021; Liu et al., 2021a): all candidate templates are evaluated using the development set and the best-performing one is chosen. This requires human annotators and does not align well with realistic zero-shot learning scenarios in which no human-annotated data is available. To address this problem, we propose a truly annotationfree perplexity-based template selection method for zero-shot prompt learning: Perplexity Selection (Perplection). Experiments show that Perplection is highly likely to select the most effective template accommodating true zero-shot scenarios.
In this paper, we first describe cloze-style prompt learning and corresponding terminologies in Section 2. Then, in Section 3, we present our hypothesis that underpins the work. Based on this hypothesis, in Section 4 we detail Perplection that uses perplexity to select templates a priori without the need of any annotated examples. Section 5 describes a pilot study and in Section 6, we present realistic experiments that show that Perplection leads to performance on par with other zero-shot prompt methods that utilise a development set. Finally, Section 7 discusses the underlying rationales and the potential impact of the work in a large language models (LLM) era.
To the best of our knowledge, we spearhead the performance screening of prompt templates for a realistic zero-shot text classification without using any human-annotated data.*

heading: 2 Preliminaries
text: In this section, we describe basic concepts and terminologies associated with prompt learning.

heading: 2.1 Prompt Learning
text: Note that the prompting settings and terminologies used in this work are mainly derived from the work that focuses on manual/automatic cloze-style discrete templates (Gao et al., 2021; Schick and
*Code is available at https://github.com/ GeorgeLuImmortal/Perplection_ACL2023.
Schütze, 2021a,b; Tam et al., 2021). As text classification is well studied in prompt-based learning tasks (Liu et al., 2021a), we use a simple binary sentiment analysis task to demonstrate zero-shot prompt learning in our work. Specifically, given an input text x, for example “I love this movie.”, we are interested in classifying the sentiment polarity, y, of this input text, i.e., ++ for positive or −− for negative. The cloze-style prompt method modifies the input x and output y to further exploit the capabilities of pre-trained language models. Formally, we first manipulate input text x to construct a new input text, x′, by prefixing (or suffixing) x with a template text sequence, t, that includes a “[MASK]” token. So, x′ = [x, t] or x′ = [t, x]. For example, if we have an input x =“I love this movie.” and we decide to prefix a template t =“Overall, it was a [MASK] movie.”, x′ will become “Overall, it was a [MASK] movie. I love this movie.”.
Next, x′ is fed into a language model to predict the likelihood with which different tokens fill “[MASK]”. This can be achieved by applying an MLM head. Usually, researchers use prior knowledge to limit the set of potential filled tokens to those relevant to the task of interest. For example, in the sentiment classification example only two tokens would be considered: “good” and ‘bad”. We call each of these a label word, w, (Liu et al., 2021a). Finally, we define a mapping function (or verbaliser) (Liu et al., 2021a), v, to reverse the predicted label word back to the target y, for example {good:++, bad:−−}. In this way the prompting method unifies a binary classification objective into an MLM objective, reusing a MLM head to perform zero-shot prediction.

heading: 2.2 Language Discrepancy and Objective Gap
text: Previous research (Liu et al., 2021a) has shown that prompt learning can help pre-trained language models better adapt to downstream tasks by bridging the gap between pre-training and the downstream task. To be specific, prompt learning allows pretrained language models to take on a greater role
in prediction, rather than just extracting features. In light of the above finding, we identify two obstacles to combining pre-training and a downstream task: language discrepancy and the objective gap. The objective gap describes the difference in training objectives between pre-training (e.g., next sentence prediction or MLM) and a downstream task (e.g., sequence classification or sequence labelling). Language discrepancy refers to the linguistic differences between a pre-training corpus and downstream datasets, including different vocabularies, word frequencies, syntactic arrangements, etc.

heading: 3 Hypotheses
text: This section proposes two hypotheses that underpin our work, and describes the way they interpret observations in the literature.

heading: 3.1 Hypothesis I: Cloze-style Prompting Offers a Better Feature Space
text: Our first hypothesis is that the use of a cloze-style prompt in text classification alters the input data distribution in a way that encourages the input data to be more effectively represented in a new feature space. To illustrate this, Figure 2 presents a UMAP (McInnes et al., 2018) visualisation of a sentiment analysis dataset, WEIBO, with and without prompt templates. It is obvious that after being prompted with a task-specific template, “[very/not] pleased.”, data from different classes is much better separated within the resultant feature space (Figure 2(b)) than when no prompt template is used (Figure 2(a)). This shows that a pre-trained language model can inherit zero-shot capabilities when given appropriate prompts, even without using any humanannotated examples.
So how do pre-trained language models construct such effective feature spaces? We conjecture that this is because some knowledge of downstream tasks has been implicitly encoded into models through pre-training (e.g., MLM for encoderonly model or Next Word Prediction for decoderonly models). Prompt learning finds a method to uncover the knowledge obtained in pre-training. Therefore, in this paper, we refer to this feature space as the “pre-trained feature space”.

heading: 3.2 Hypothesis II: Language Discrepancy Measures the Efficacy of Prompting
text: Additionally, we aim to understand what makes a template effective at forming a useful pre-trained
feature space. We believe that the difference in language between pre-training corpora and downstream datasets after prompting can be used to assess the effectiveness of templates.
Figure 2(c) shows an example. When the text inputs are given a prompt that is unlikely to be used in sentiment analysis texts, “[yellow/green] black.”, the data from different classes is not well separated in the feature space (as compared to Figure 2(b)). We believe that this is because models rarely encounter the text “yellow black” or “green black” prefixed in a sentiment-bearing text in the pretraining corpora, and that this language discrepancy limits the model’s ability to effectively represent the data. In contrast, expressions like “[very/not] pleased.” (Figure 2(b)) are often used in context related to emotions and therefore appear more frequently together with sentiment-bearing text in the pre-training corpora. This makes it easier for the model to form a useful pre-trained feature space.
Broadly speaking, we suppose that the objective gap has been greatly reduced by reformulating the downstream task to use a prompt in text classification. The inconsistency is largely due to the language differences between the pre-training data and the downstream data. Using prompt templates helps to align the downstream text with the text in a pre-training corpus with respect to language discrepancy. The smaller the language discrepancy between the pre-training data and the downstream data that are being prompted, the more likely it is that the data will be represented well in the feature space, resulting in better zero-shot performance.

heading: 4 Method
text: As discussed in Section 3, a heuristic approach can be employed to select the most effective templates in zero-shot text classification. One way to do this is to utilise language discrepancy to “forecast” the performance of different prompt templates. Specif-
ically, the prompt template that results in the lowest language discrepancy when prefixed to a given input text can be considered the most effective. However, how can the language discrepancy between downstream text and pre-training corpora be measured? In this study, we propose using perplexity (Brown et al., 1992) as an approximation of language discrepancy.
Perplexity is one of the most common metrics for evaluating language models, and is defined as the exponential average negative log-likelihood of a sequence:
PPL(x) = exp { −1
t
t∑
i
log pθ (xi | x<i) }
(1)
where x = [x1, x2, ..., xt] is a tokenised text sequence; and log pθ (xi | x < i) is the loglikelihood of the ith token conditioned on the preceding tokens x < i computed by a language model. Intuitively, given a certain language model, lower perplexity for a corpus of sentences indicates a model is familiar with that corpus. Basically, the language model with the lowest perplexity is chosen as the most reliable proxy for modelling the distribution of the pre-training corpus.
Analogously, we assume that prompt templates resulting in low perplexity when prefixed to a given input are likely to be effective templates, eliminating the need for a human-annotated development set, which is required in most previous work (Liu et al., 2021a; Lester et al., 2021; Gao et al., 2021). Specifically, as shown in Figure 1, we prefix original input x with various prompt templates to form new prompted texts. For each template, since we
have two label words (i.e., “very” and “not”), one original input x will generate two prompted texts (i.e., “Very pleased. Such a bad movie!” and “Not pleased. Such a bad movie!”). Then we compute the mean perplexity score of these two prompted texts as the score for the template. Finally, the template (where the label words will be replaced with "[MASK]" token) with lowest score is selected to be prefixed to the original input, constructing new input x′ (i.e., “[MASK] pleased. Such a bad movie!”) to perform a zero-shot prediction. This is quite different from previous methods with datasetspecific (Gao et al., 2021; Sun et al., 2022) or classspecific templates (Zhou et al., 2022). We refer to the method as Perplexity Selection (Perplection).

heading: 5 Pilot Study
text: The aim of the pilot study described in this section was to qualitatively validate the hypotheses proposed in Section 3, and to examine the utility of perplexity as a metric for screening prompt templates (another study that examines the utility of perplexity is presented in Appendix D). To this end, we manually curated four prompt templates as shown in Table 1. We then analysed the perplexity and zero-shot performance of each template, seeking to determine whether there is a correlation between perplexity and zero-shot performance.

heading: 5.1 Datasets
text: We conducted the pilot study using four publicly available Chinese sentiment analysis datasets from various domains. These datasets are: DOUBAN, a movie review dataset; WEIBO, a social media comment dataset; WAIMAI, a takeaway comment
dataset; ECOMMERCE, an e-commerce dataset.

heading: 5.2 Perplexity
text: We use the Chinese RoBERTa model* as the backbone pre-trained model. Given a pre-trained language model, we use it to compute the mean perplexity of downstream datasets that are being prompted, to approximate the language discrepancy. That is, lower perplexity indicates smaller language discrepancy between the pre-training corpus and the prompted downstream dataset.
Note that perplexity, as originally defined, applies specifically to causal language models (i.e., autoregressive language models). As suggested in previous work (Liu et al., 2019; Salazar et al., 2020), perplexity for bidirectional models like BERT/RoBERTa can be made analogous to that for causal language models by replacing log pθ (xi | x < i) with log pθ (xi | c) in Equation 1. Here, c refers to the context text, which is the whole sentence except for the ith token. This suggests that the perplexity of each token is not only conditioned on the preceding tokens but also the succeeding tokens. We added a template to each example, replaced the “[MASK]” with label words from the prediction problem, and calculated the average perplexity for each example. We then averaged the perplexity scores of all examples to get the overall perplexity of the dataset.
During preliminary experiments, however, we found that this definition of perplexity has the drawback of favouring longer sentences. That is, a sentence is assigned a lower perplexity, not because the pre-trained language model is more able to model this sentence (i.e., low language discrepancy), but rather because the text is longer. We conjecture that this is due to the penalty term in Equation 1 that divides the sum of log-likelihood by the sequence length t. The detail of our preliminary experiments regarding perplexity are provided in Appendix A. The focus of this pilot study, however, is to illustrate the impact of language discrepancy rather than finding useful measures of perplexity. So, to mitigate against the drawbacks of the perplexity definition the four datasets used in our experiments were subsampled to include only sentences with between 14 and 15 words, as well as to enforce a 50:50 class balance. Also, all hand-crafted templates have similar lengths (in Chinese).
*Available at https://huggingface.co/hfl/ chinese-roberta-wwm-ext.

heading: 5.3 Zero-shot Result Analysis
text: The accuracies achieved using different prompt templates for four datasets are shown in Table 1. These results demonstrate that prompt learning can equip a pre-trained language model with zero-shot capability when proper templates are provided. However, the performance of Template 4 (i.e., “[yellow/green] black”) demonstrates that “unusual” prompting (i.e., texts that models are unlikely to see during pre-training) has limited contribution to zero-shot prediction, which is consistent with our expectation.
To conclude, the results of the pilot study verify our hypothesis that in prompt learning, task-related templates are more useful in shaping a good pretrained feature space. The big difference between zero-shot performance across different prompting approaches in the pilot study shows that it is crucial to search for ideal prompt templates in prompt learning. We argue that this problem can be addressed by using perplexity as discussed in the following subsection.

heading: 5.3.1 Perplexity Analysis
text: Table 1 also conveys a very clear message that as perplexity goes up, the zero-shot performance becomes worse. For example, the perplexity of Template 1 decreases from 24.61 (DOUBAN), to 19.78 (WEIBO), to 16.44 (WAIMAI), to 13.71 (ECOMMERCE); while the zero-shot accuracy consistently increases from 57.12 (DOUBAN), to 61.79 (WEIBO), to 67.80 (WAIMAI), to 73.12 (ECOMMERCE). This pattern can also be observed for Templates 2 and 3. Furthermore, when comparing sentiment-bearing templates (Templates 1-3) to the sentiment-irrelevant template (Template 4) across datasets, it is evident that the sentimentirrelevant template consistently yields the highest perplexity and the lowest accuracy. The experimental results can partially verify our hypotheses that as the language discrepancy decreases (i.e., lower perplexity), it is easier for prompts to align downstream data to a pre-trained feature space. The next section describes experiments that show how the Perplection approach takes advantage of this.

heading: 6 Experiments
text: In this section, we demonstrate the proposed Perplection approach in a more realistic and useful experimental setting to verify whether we can use language discrepancy to forecast the efficacy of
prompt templates for zero-shot classification.

heading: 6.1 Datasets
text: In addition to the datasets mentioned in Section 5.1, we also utilise four text classification datasets from the FewCLUE benchmark (Xu et al., 2021): EPRSTMT (e-commerce comment sentiment analysis), CSLDCP (scientific literature subject classification), TNEWS (news classification), and IFLYTEK (APP description topic classification). To evaluate whether Perplection can be extended to other languages, we also evaluate Perplection on three English datasets: SST-2 (sentiment analysis) (Wang et al., 2018), TweetEval (hate speech detection) (Barbieri et al., 2020), and AG News (multi-class topic classification) (Zhang
et al., 2015). Note that in contrast to the pilot study, in these experiments we did not subsample the datasets to make their sentences the same length.

heading: 6.2 Setup
text: All manually crafted templates are presented in Table 4. All the verbalisers and manual templates for English datasets can be seen in Appendix C. We perform Perplection based on these manually designed templates (MPerplection). If perplexity is an ideal metric, the performance of this method will be better than random template-example matching (MRandom). We then construct a more aggressive setting where templates are generated automatically by LM-BFF algorithm (Gao et al., 2021) (more detail is included in Appendix B) and ap-
ply similar template selection procedures to those described for manually crafted templates. These are dubbed APerplection and ARandom. In order to obtain a robust assessment of the random variants, we conduct five independent runs of the experiments using different random seeds and report the average results. Note that both manually crafted and automatically generated templates are constructed to have similar lengths.
We report the results based on both RoBERTa and BERT* to demonstrate the proposed method is agnostic to the pre-trained model used. We also report the performance of another two state-ofthe-art zero-shot prompting-based methods: NSPBERT (Sun et al., 2022), and Zero-PET (Schick and Schütze, 2021a; Xu et al., 2021). They are strong baselines whose settings comply with the corresponding work (further implementation details are provided in Appendix C).

heading: 6.3 Results
text: Comparison to random baselines: The results of the Perplection variants and their corresponding random counterparts were compared in Table 2. It can be seen that when using manually crafted templates with both BERT and RoBERTa, Perplection was able to actively select more useful templates compared to the random selection, as indicated by the significant improvement in performance (MRandomB vs. MPerplectionB and MRandomR vs. MPerplectionR). Also, when using automatically generated templates, Perplection is able to choose more effective templates, particularly when using RoBERTa (ARandomR vs. APerplectionR). These findings suggest that the templates selected by perplexity are more useful and deliver better performance. However, results also show that Perplection is less effective when automatically generated templates are used, which will be discussed in the next section.
Manual templates vs. automatic templates: Table 2 shows that variants using manually generated templates outperform their counterparts using automatically generated templates. We conjecture that the poor quality of automatically generated templates may hinder the performance of Perplection. In other words, the pool of automatically generated templates may be insufficient in diversity for Perplection to have an impact.
*https://huggingface.co/bert-base-chinese
As illustrated in Table 4, the majority of automatic template texts display minimal variations and lack coherence, which is in stark contrast to the manual templates. In this case, templates tend to generate similar perplexities, leading to little distinction between them based on perplexity. To illustrate this, we report the standard deviation of perplexity for both manual templates and automatic templates in Table 5. It can be observed that for all datasets, the standard deviation of perplexity for manual templates is higher than that of automatic templates, showing that perplexity is more useful when the templates are of higher diversity.
It is suspected that the quality of the automatically generated templates is constrained by the capacity of the pre-trained T5 model. We believe that this can be improved by changing the T5 backbone or resorting to other methods that automatically generate templates using annotation information (Lester et al., 2021; Liu et al., 2021b; Li and Liang, 2021; Liu et al., 2022b). We leave these explorations for future work.
Comparison to state-of-the-art approaches: We compare our best performing method (MPerplectionR) with other state-of-the-art zero-shot methods, results are shown in Table 3. We find that the performance of Perplection consistently surpasses Zero-PET for all datasets by a large margin except for TNEWS, and is competitive with NSP-BERT in some datasets such as DOUBAN (60.74 vs. 60.85). Note that both Zero-PET and NSP-BERT used a human-annotated development set to select the most suitable templates while Perplection does not require any annotated data.
For the IFLYTEK dataset, Perplection seems less competitive as compared to Zero-PET and NSPBERT. Specifically, the latter two methods heavily rely on the post-hoc selected template “This
is a [MASK] app.” (see Appendix C) with the development set quite close to target domain of interest, whereas Perplection has more generic templates (in Table 4, those prompts are task-related but not domain-relevant). Thus, the suboptimal performance of Perplection can also be explained by our hypothesis that generic templates are less effective at aligning the downstream data into a pre-trained feature space compared to those finegrained domain-specific templates. We suspect that this can be addressed by providing Perplection with several domain-related fine-grained templates to select from. We leave these explorations for future work. All observations, however, show that it is effective to use perplexity to rate templates and select desired ones accordingly.
Results on English datasets: Table 6 compares the performance of Perplection to random baselines on three English datasets. Perplection consistently tops the comparison in almost all cases except for SST-2 with RoBERTa. This observation supports the supposition that Perplection is agnostic to the pre-trained model used, and shows that it is promising to extrapolate results to other languages.

heading: 6.4 In-depth Analysis
text: We conduct an in-depth analysis based on MPerplectionR. For brevity, we apply each manual prompting setting to all examples from the four datasets (i.e., DOUBAN, WEIBO, WAIMAI, ECOMMERCE) and aggregate the accuracy score as a post-hoc measurement of template quality. For each template, we also compute its frequency of being selected. The results are presented in Figure 3. It shows that templates with lower perplexity are more likely to achieve better performance. To be specific, there is 60% chance for Perplection to select the second best performing template (i.e., “[MASK] fond of it.”) and around 10% chance to select the best performing template (i.e., “[MASK] satisfied.”). For templates with no discriminative ability e.g., “[MASK] good.” and “[MASK] ok.”, our method has almost no chance to select them. Most importantly, the selection based on perplexity is annotation-agnostic and allows us to “foresee” the result to some extent without the need of a human-annotated development set. To conclude, the results demonstrate that perplexity is a reasonable metric for evaluating prompting settings.

heading: 7 Discussion
text: What contributes better zero-shot learners? This work empirically reveals that the large language discrepancy between the pre-training corpora and the downstream data may hinder the zeroshot generalization. On top of that, we develop a perplexity-based scheme that leverages cloze-style prompt templates to bridge language discrepancy and thus, fully releases the potential of pre-trained language models. The significance of this work lies in its pioneering study of a feasible objective for optimising REALISTIC zero-shot prompting templates. The idea may be applied to various variations (e.g., continuous prompts) beyond the discrete prompts currently being studied.
Why REALISTIC zero-shot matters? In this work, we constantly emphasise a realistic zero-shot scenarios (no labelled data), as opposed to the existing zero-shot setting in the field of NLP (Xu et al., 2021; Sun et al., 2022) or Multi-modality (Radford et al., 2021), where a development set is available for template selection or hyper-parameter tuning. Realistic zero-shot can be quite appealing for industrial scenarios and thus, this research opens up a new avenue for research in the field of zero-shot learning, probably inspiring follow-up studies in broader tasks for advancing the zero-shot learning in industrial applications (especially in many low-resource scenarios).
Potential impact in the LLM era. In light of the advancements in large language models (LLM) based on the decoder-only architecture (Zhao et al., 2023), searching for effective instructions or incontext demonstration examples (Zhang et al., 2022) has become an essential challenge. Perplection can be seamlessly applied to decoderonly models for searching effective instructions/incontext examples for various natural language gen-
eration (NLG) tasks. We make our code available for replication and further extension to NLG tasks by the community.

heading: 8 Conclusion
text: We developed Perplexity Selection Prompt (Perplection) a method that enables real-world zeroshot text classification without the use of any human-annotated data. A pilot study demonstrated that Perplexity can be an effective measure of the efficacy of templates. Experimental results show that, for datasets in both English and Chinese, our method can boost zero-shot performance of clozestyle prompt learning in binary sentiment analysis as well as multi-class classification, without using a development set. Further in-depth analysis supports the observation that Perplection can “foresee” the efficacy of prompt templates.

heading: 9 Limitations
text: In this study, we mainly utilised the BERT family of models for Chinese text classification tasks. Given the similarity with respect to transformer language models and pre-training paradigms, as well as the preliminary results on English datasets as discussed in Section 6.3, we may be able to extrapolate the results to other architectures/tasks/languages.
For example, Perplection can be seamlessly apply to decoder-only models (e.g., GLM (Du et al., 2022), LLaMA (Touvron et al., 2023)) to see whether it can boost the performance for those NLG tasks. But further investigation is needed to verify the utility of findings on other model architectures, tasks, and languages. In the future, we expect to see Perplection applied to different NLG tasks such as seq2seq information extraction (Lu et al., 2022b), question answering, arithmetic reasoning, machine translation or even multi-modality tasks.
Also, utilising Perplection may exacerbate the inherent limitations of pre-trained language models. We suspect that, in instances where the model has not been exposed to certain texts or concepts during pre-training, reliance on perplexity for template selection may result in subpar performance. In the future, we expect to explore whether we can alleviate this problem by certain annotation-free methods, such as continuous self-supervised training with downstream data, or extend our method in a few-shot setting where limited label information is available.
Besides, the use of perplexity as a metric has the drawback of favoring long texts, which forces us to design templates of the same length. Therefore, a length-agnostic metric can be considered as an alternative.

heading: 10 Ethics Statement
text: We honor the ACL Code of Ethics. No private data or non-public information was used in this work. We conducted our research in an objective and unbiased manner. We take full responsibility for the content of this paper and stand behind the accuracy and integrity of our work.

heading: Acknowledgements
text: We would like to thank anonymous reviewers for their insightful comments to help improve the paper. This publication has emanated from research conducted with the support of SenseTime Research.

heading: B Automatic Template Generation
text: Similar to Gao et al. (2021), for the DOUBAN, WEIBO, WAIMAI, and ECOMMERCE datasets we fix the verbaliser to {very: ++, not: −−}, and use T5-v1.1-base-chinese* to automatically generate templates. Specifically, Gao et al. (2021) assume a
*https://huggingface.co/uer/ t5-base-chinese-cluecorpussmall.
few-shot scenario using ground truth label word as well as corresponding examples to generate a number templates. They then sort generated templates based on the aggregated generation probability (the calculation of generation probability also needs label information) of the whole training set. However, our experiment assumes a zero-shot scenario with no labelled data. Thus, for each dataset, we first randomly sample 50 examples from the pool. For
each example, we use label words indicating both sentiments to generate templates, one for each sentiment, resulting in 100 templates in total. Then we remove duplicate templates, leaving around 59-73 templates remain per dataset respectively.
For the EPRSTMT, TNEWS, CSLDCP, and IFLYTEK datasets, whose automatically generated templates have been made available,*, we directly use those existing generated templates. We remove duplicate templates and around 11-22 templates remain per dataset. All automatically generated templates can be seen at URL masked for anonymous review.
*https://github.com/CLUEbenchmark/FewCLUE/ tree/main/baselines/models_pytorch/LM-BFF/my_ auto_template.
C Implementation Details
In the implementation of Zero-PET, we use the pretrained Chinese-RoBERTa-wwm-ext model, which is identical to the model employed in Perplection. For NSP-BERT, we use google BERT-Chinese.* Templates and label words for both baselines follow the best-performing setting reported in (Sun et al., 2022; Xu et al., 2021), as shown in Table 9. The manual generated templates (in Chinese) for Perplection are also shown in Table 9. A conversion is conducted to map class names to label words following (Xu et al., 2021) to ensure all prefixed texts have similar length, as shown in Table 8. For the CSLDCP and IFLYTEK datasets we randomly subsample 15 classes to facilitate the experiments.
In the implementation of English Perplection and its random counterparts, we use the pre-trained BERT-base-uncased* and RoBERTa-base* models. Templates and label words for English experiments are shown in Table 10. All experiments are conducted on a Tesla V100 GPU with 32GB memory.
*https://huggingface.co/bert-base-chinese. *https://huggingface.co/bert-base-uncased *https://huggingface.co/roberta-base

heading: D Reverse Label Words
text: To briefly verify whether perplexity can be used to measure the quality of prompting, we perform a very simple experiment where we compute the mean perplexity score of prompted input x′ with “[MASK]” filled by ground truth label words for each dataset (called PPLg ). Then we reverse the label words filled in previous input examples (e.g., we change “very pleased.” to “not pleased.” in a positive sentiment example) and recompute mean perplexity score (called PPLr). Note that this experiment is based on RoBERTa. The results of this are shown in Table 11.
First, we notice that in Setting 1 (i.e., “[very/not] pleased.”), the mean perplexity of PPLg is always smaller than that of PPLr by a clear margin which is encouraging. This shows that the pre-trained model can perceive the change of semantics in texts. When we see the perplexity of Setting 2 (i.e., “[yellow/red] black.”, we find out the magnitude of change is much smaller, which demonstrates that replacing label words makes almost no difference to models if domain-irrelevant prompting is applied.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section 9 Limitations
3 A2. Did you discuss any potential risks of your work? Section 9 Limitations
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract and Section 1 Introduction
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? Left blank.
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 5.2 Issue of Perplexity, Section 6.2 Setup, Appendix C Implementation Details
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 5.2 Issue of Perplexity, Section 6.2 Setup, Appendix A Issue of Perplexity, Appendix B Automatic Template Generation, Appendix C Implementation Details,
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 6.3 Results, Section 6.4 In-depth Analysis
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 5.2 Perplexity, Section 6.2 Setup, Appendix A Issue of Perplexity, Appendix B Automatic Template Generation, Appendix C Implementation Details,
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: In the future, we expect to see Perplection applied to different NLG tasks such as seq2seq information extraction (Lu et al. , 2022b), question answering, arithmetic reasoning, machine translation or even multi-modality tasks. Also, utilising Perplection may exacerbate the inherent limitations of pre-trained language models. We suspect that, in instances where the model has not been exposed to certain texts or concepts during pre-training, reliance on perplexity for template selection may result in subpar performance. In the future, we expect to explore whether we can alleviate this problem by certain annotation-free methods, such as continuous self-supervised training with downstream data, or extend our method in a few-shot setting where limited label information is available. Besides, the use of perplexity as a metric has the drawback of favoring long texts, which forces us to design templates of the same length. Therefore, a length-agnostic metric can be considered as an alternative