Are BERT Families Zero-Shot Learners? A Study on Their Potential and Limitations

Abstract: Starting from the resurgence of deep learning, language models (LMs) have never been so popular. Through simply increasing model scale and data size, large LMs pre-trained with self-supervision objectives demonstrate awe-inspiring results on both task performance and generalization. At the early stage, supervised fine-tuning is indispensable in adapting pre-trained language models (PLMs) to downstream tasks. Later on, the sustained growth of model capacity and data size, as well as newly presented pre-training techniques, make the PLMs perform well under the few-shot setting, especially in the recent paradigm of prompt-based learning. After witnessing the success of PLMs for few-shot tasks, we propose to further study the potential and limitations of PLMs for the zero-shot setting. We utilize 3 models from the most popular BERT family to launch the empirical study on 20 different datasets. We are surprised to find that a simple Multi-Null Prompting (without manually/automatically created prompts) strategy can yield very promising results on a few widely-used datasets, e.g., $86.59\%(\pm0.59)$ accuracy on the IMDB dataset, and $86.22\%(\pm2.71)$ accuracy on the Amazon dataset, which outperforms manually created prompts without engineering in achieving much better and stable performance with the accuracy of $74.06\%(\pm13.04)$, $75.54\%(\pm11.77)$ for comparison. However, we also observe some limitations of PLMs under the zero-shot setting, particularly for the language understanding tasks (e.g., GLUE).

Reviews and author responses:

 Dear Reviewer S18q,

Thanks for your further comments. 

We do honestly reflect what we are thinking of this work. We agree that the motivation of the strategy is not so clear to present the spirit behind it. But we still would like to clarify the motivation of this work and the contribution of our work.

1. As we replied, we aim to provide a comprehensive study, simple multi-null prompt strategy/expanding vocabulary set, coarse-to-fine studies, and the observed limitations of BERT-family pre-training models in zero-shot learning. Instead of the simple strategies only. 
2. We conduct extensive experiments of different studies to show the potential and limitations of the BERT-family pre-training models with different prompt strategies. We are not selling the Multi-Null Prompt strategy as our core motivation, instead, we just want to show this kind of simple strategy can already achieve good performance. 
3. This kind of simple strategy can be viewed as a simple "trick", though the motivation is not so clear, the empirical study is important since it is indeed effective, which can have a realistic and practical benefit to real zero-shot scenarios, this kind of easy and hand-free help should be considered. 
4. We would like to discuss that, even if we do not have further clear understanding of the methods, to point out this kind of observation so that to motivate more people to understand it is still a contribution to some extent. This can help inspire other researchers to better study the knowledge behind it. 

Thanks again, we do hope that the impact is not considered simply from the novelty or improvement space.  

The paper conducts a theoretical study of zero-shot learning in BERT-style pre-trained LMs over 20 datasets. They test different methods of prompting as well as repurposing the MLM head classifier. They propose a different prompting method, Multi Null Prompt, which is a simple method where multiple MASK tokens are inserted at random locations in the prompt and the outputs with respect to the multiple MASK tokens are aggregated. Multi Null Prompt seems to perform better than the other prompting methods. They also propose to incorporate more similar words to the label words (cosine similarity to the label words used for zero shot learning), which also helps. They do some experimental analysis of MASK tokens in Multi Null Prompt, including their position, whether other tokens can be used instead of MASK, different ways of aggregating the outputs from the multiple MASK tokens.  The paper also tries GLUE tasks and shows that zero shot performance on these tasks is comparatively worse. Strengths
- The paper conducts a thorough empirical study with many datasets and many methods. The results are pretty intuitive - the prompting methods are better than "tuning -based" methods, engineered manual prompts are better than non-engineered. 
- The proposed Multi Null prompting and label word search seem to improve performance steadily and the approach is simple.
- After proposing Multi Null prompt, the paper empirically analyzes the components of the method, showing why certain decisions were made.
- Improving prompting methods is an important question, although it's unclear why exactly we want the zero shot setting if few shot gives substantial benefits and its not very hard to get a few examples.


Weaknesses
- It's unclear why / with what insight from the experiments the new strategy of Multi Null Prompt was proposed - there isn't a clear issue that comes out of the general empirical study -- Multi Null Prompt just seems to appear without much motivation. More motivation / delineation of a concrete problem that comes from observations in the empirical study could make the logic of the paper more clear.
- In Table 1, it seems that only some of the results have error bars?
- In Table 3, it's unclear how the alternative words for MASK were chosen. Are they random words? It's not quite clear what the purpose of this experiment is, though it is an interesting exploration. The word to use for Multi Null Prompt doesn't seem like an obvious hyperparameter to ablate. Overall, the results show that other words may be worse or similar to using MASK, so we may as well use MASK.
- One would hope that in Fig 2, adding more similar words helps more, but this isn't really the case. Perhaps it would help to show how much it improves initially from zero additional words.
- As mentioned in a footnote, Multi Null Prompt cannot be used with some types of tasks that need multi MASK positions. Is there a proposal for how to deal with this limitation? 

Questions
- In the intro, why is zero shot referred to as a more general setting than few shot? It seems the other way around.
- In Fig 1, interestingly after about 3 MASK tokens, the performance tends to go down for Multi Null prompt. Any intuition for this? 
- In Table 4, why does ALBERT have such a different behavior on IMDB? 
- In Fig 2, what does number 1, number 2 number 3 mean?  The paper provides an empirical study of BERT models for zero shot learning with interesting results, including two simple proposed methods that improve performance. The motivation for the multi null prompt method isn't very clear, and some of the experiments were not completely motivated, but overall it seems like a good empirical study.

 Thanks for the response. I think the contributions of multi-null prompting and using the expanded vocab set are nice tools that seem to be complementary when added to other methods. Although the authors give an account of how they came up with Multi Null prompt, I still think the motivation for multi-null prompt isn't clear, or at least not very principled. It's unclear to me why such a thing should work, and I feel that this could affect the impact. Given that the empirical methods aren't the most novel, some more understanding of the methods would make the work more complete. I've updated my score to reflect this room for improvement. 

 Dear Reviewer 9k3x,

Thanks for your further valuable comments! We thank you for the important “baselines” you suggested again, and we would like to provide the following discussions to your additional comments from different views. 

We first provide the discussion about the two “baselines” you suggested. 

* We totally agree that Expanded Vocabulary Set can hugely boost the results. This can also be proven from Table 1 we provided before (Null Prompt*, Multi-Null Prompt*). What we provided is a simple strategy about expanding vocabulary set by searching similar label words by embedding cosine similarity, instead of the previous methods that leverage external knowledge/unlabeled data/validation data to expand label set. Therefore, this simple strategy is more practical in zero-shot learning scenarios and is one of the contributions of our work. 
* The improved performances of Manual Prompt# + Expanding Vocabulary Set and Manual Prompt (Prior) + Expanding Vocabulary Set demonstrate the importance of expanding vocabulary set. Since we are introducing expanding vocabulary set as a simple strategy, the good performance of “expanding vocabulary set” on the manual prompt setting can be shown as a demonstration of the effectiveness of our expanded vocabulary set strategy. We want to kindly mention that this setting (expanding vocabulary + manual prompt) is **not regarded as a baseline but a verification experiment**, since **we are not claiming** Multi-Null Prompt + Expanding Vocabulary Set is the best strategy that should beat other all settings.
* As for Multi-Null Prompt strategy, (1) if we compare Manual Prompt (Prior) + Expanding Vocabulary Set and Multi-Null Prompt + Expanding Vocabulary Set, the results are also comparable as you mentioned. But please note that the Manual Prompts (Prior) are carefully selected by validation sets, which is not practical under the true zero-shot setting. This is verified by [1] that a large validation set helps a lot to select good prompts, but requiring a large validation set is not feasible in zero-shot learning. Therefore, as Figures 1 shows, because Multi-Null Prompt is not sensitive to the [MASK] number, it can perform well even without the help of the validation set, which is **a more practical method under the true zero-shot settings**. (2) Besides, Multi-Null Prompt + Expanding Vocabulary Set can outperform Manual Prompt# (without being selected carefully by validation sets) + Expanding Vocabulary Set, which can again demonstrate the simple Multi-Null Prompt is effective compared to human efforts required manual prompts (without careful selection by validation sets). (3) Further, the second experiment you suggested (Manual Prompt + Multi-Null Prompt) is more effective than Manual Prompt only, this also shows Multi-Null Prompt is beneficial. We agree that this can be a more principled approach, but not too appliable in true zero-shot learning settings, we would include these results in the later paper version. 
* Following with above points, our proposed “Multi-Null Prompt” and “Expanding Vocabulary Set” are two strategies that can be simply and practically applied in zero-shot setting. What we want to show is that these two strategies can be effective to significantly achieve performance improvement for zero-shot learning, no matter used together (Multi-Null Prompt + Expanding Vocabulary Set) or separately applied with other prompt methods (e.g., manual prompt + Expanding Vocabulary Set (you suggested), or manual prompt + Multi-Null Prompt (you suggested)). Again,**these are not “baselines” that we need to beat**, which are instead settings/extensions we can verify the effectiveness of these two strategies, and we want to thank you for your suggestions. As shown in our paper title, these are showing more “potential” of BERT-family models as zero-shot learners, which should be more studied in the future. 
* We want to kindly clarify that “This is an important missing baseline because it would affect the paper’s conclusions (about methods requiring no human input and auxiliary data) if any of the representation aggregation approaches with the expanded vocabulary set outperform *-NULL* approaches”. **We never claim** that the combination of our proposed two strategies (Multi-Null Prompt, Expanding Vocabulary Set) which requires no human input and auxiliary data is the best strategy that should output perform other strong/good baselines, e.g., Manual Prompt (Prior), KPT. We just want to show this simple and human-free strategy combination can already achieve good performance. It is clear that this combination is not the SOTA performance. 



 * As for the last point, we also want to clarify the target of our paper. We focus on providing a comprehensive empirical study of the BER- family under the zero-shot setting and we propose some practical strategies that do not need human effort, validation set, or unlabeled data and can be combined to various baselines, which all have potential and can also be helpful for the community to explore the zero-shot settings further. In a short summary, **the comprehensive study, simple multi-null prompt strategy/expanding vocabulary set, coarse-to-fine studies, and the observed limitations are the contributions**.

And for the structured multiple tags in your further comment, we further do experiments according to your suggestion on IMDB to test the multiple tags prompt. The prompt with multiple tags and the results are as follows:

Templates  | 1-MASK | 2-MASK | 3-MASK | 4-MASK | 5-MASK | 6-MASK | 7-MASK | 8-MASK
--|---|---|---|----|---|----|----|------
[X] In summary, the film is [MASK]. | 93.68|93.87|94.00|93.80|93.32|92.88|92.40|92.15
[X] In summary, the film is [MASK] and [MASK] but also [MASK]. |86.77|86.92|90.58|91.29|91.50|91.47|91.37|90.99


We can see that the multiple tags prompt is, unfortunately, worse than the conventional manual prompts with multiple [MASK]s. Combined with multiple [MASK]s, the multiple tags prompt’s accuracy is ranged from 86.77 to 91.50, which is still worse than the manual prompt without multiple [MASK]s (e.g., 93.68 in above line). We will deeply study why multiple [MASK]s can work in future studies.

Therefore, we want to kindly mention that the efforts we paid are based on your great suggestions and intuitions, which can help further polish our paper with more extensive experiments to show the effectiveness of these simple strategies. We kindly hope that you could reexamine our paper. 


Reference:

[1] True Few-Shot Learning with Language Models, https://arxiv.org/pdf/2105.11447.pdf

 It is appreciated that the authors took the time to conduct the experiments that I suggested and I thank them for their hard work.

**[Using expanded vocabulary set for baselines]** As I suspected, it seems that a huge part of the boost in the results is coming from the expanded label set and the other baselines see great improvement when this is done. Since the results of adding the expanded label set with the baselines are comparable (and it seems even better - 79.59 vrs 77.14), it  calls into question the strength of the method as an alternative to the baseline given that the baseline (in my opinion) present a slightly more principled (and potentially more interpretable) way of solving the problem.

**[Prompts with multiple masks]** Again as I suspected, using prompts with multiple masks also leads to improved results. I therefore do not think the MULTI-NULL approach is that much of a leap from conventional prompting since prompts can always be structured to require multiple tags eg ‘In summary the film was — and —  but also — “ 


Thanks once again for updating the paper - the effort is appreciated. Based on the results of the experiments and given that they confirm my intuitions about the method - I will not be raising my score.

 6.	As for the failure of NLU tasks, thanks for your provided reason. Both yours and our suspected one are all possible since there is no comprehensive analysis behind the observations currently. However, we want to respectfully point out that a surprising and different observation indeed has its own value. We show the different behaviors between the NLU tasks and the classification tasks (success in the text classification) to researchers, where this observation itself is valuable to some extent, and this can greatly inspire other researchers and us to deeply dig in and analyze this observation in the future. We agree that a more thoughtful analysis can be more impactful of this finding, and we will continue pushing it and working on more tasks.

We would like to provide the **responses to the weaknesses** as follows:

1. Thanks for providing the related works. We have cited and discussed them in the updated version. Let us have short discussions here.
   * [1] study the `true few-shot learning’ while we are on the zero-shot learning, the settings are clearly different. Besides, though both of us focus on the prompt for language models, [1] aim to test the two model selection criteria for choosing prompts while we are studying the Null Prompt-based strategies without any heuristic prompt design or prompt selection process. Further, while [1] claim ‘previous work significantly overestimated the true few-short ability of LMs given the difficulty of few-shot model selection’, we find the simple Multi-Null Prompt is surprisingly effective for text classification without any model/prompt selection.
   * As for [2], the similarity is that both [2] and ours are studying the prompt ability in the zero-shot learning tasks. But the differences are also obvious.
      * The motivation and the studying scenario are quite different. [2] focus on the zero-shot task generalization ability by multi-task prompted training, which utilizes multi-task and multiple datasets in a `training’ system, and then test the generalization ability of prompt in the zero-shot learning (new task) setting, while our study is a pure zero-shot learning setting without any training.
      * [2] leverage multiple prompts for each task (dataset) when training multi-tasks and they also propose a prompt template learning system for these large number of prompts. In comparison, we study the Null Prompt setting, and we propose a simple Multi-Null Prompt strategy, which is much easier and lightweight to be adopted in real scenarios.
      * The basic models are also distinct. [2] test models on an encoder-decoder architecture based on T5, which transforms all tasks into generation tasks. We directly test the general BERT-style pretraining models that are more well-known and widely adopted.
      * Finally, we respectfully disagree that the existence of [2] undercuts the value of our work, since [2] and our work are parallel work at a similar time point (the arxiv version is later uploaded after the ICLR submission deadline), and we both work on the prompt in zero-shot learning, this greatly demonstrates the importance and popularity of this research direction, which instead highlights the value of our work. Besides, the differences are shortly discussed in the above content, and the views in our work and [2] can be combined to further improve the zero-shot learning.

2. We would like to highlight our contributions and important findings again. 
   * The simple Multi-Null Prompt strategy is surprisingly effective for text classification. This greatly reduces the effort of prompt design/selection/tuning and shows the great potential of Null Prompt methods.
   * We give a comprehensive study from different views, including the number of [MASK] tokens, ensemble methods, other prompt words, expanded similar label words. Though some of them are not surprising, presenting the comprehensive results and a complete picture of the prompt strategy is also important to clear and understand each point of this strategy (there are also surprising findings, please see `Most Surprising Findings’ in the Introduction section).  
   * We also show the limitation of the Multi-Null Prompt strategy in the NLU tasks, and we provide several hypotheses. The different behaviors in these two kinds of tasks can also inspire researchers to think and analyze the reasons behind deeply, hence, to further improve zero-shot learning. We believe the above contents are valuable to the community. 
   * Thanks for your advice to combine other prompt methods. We conduct experiments accordingly to combine our simple Multi-Null Prompt with the manual prompt method, which is the above fourth response point (please see above), and the experimental study is put in Appendix A.2 in the updated version.

3. We put the discussion in response to the fifth summary (please see above). We expect more discussions about these studies.  


 Dear AC and Reviewers,

Thanks for your review comments and we have revised our paper draft accordingly. Here is a short summary (the changes are in blue color in the updated version). 
1. We have clarified unclear descriptions of the illustration of Figures. 
2. We have added citations and discussed the related works provided by reviewers.
3. We have added a study experiment about manual prompt + our simple multiple mask strategy in Appendix.
4. We have added several examples of different prompt strategies to better distinguish the differences.
5. Several typos are also fixed. 

Thanks and look forward to further feedback. 

 Thanks for your thorough reviews and comments on our paper.

We first want to thank you for your comprehensive summary of our paper. Here are some detailed **responses to your summary**:
1. Thanks for pointing out the related work that we missed. We have cited it in the updated version. We want to kindly mention that they are studying under the true few-short learning to show prompt selection is important, while we are under the zero-shot setting and we focus on the simple Null Prompt based methods. Therefore, even for the same observation of prompt sensitivity, they come from different views, and we attempt to give some inspiration to zero-shot learning research.
2. Yes, our goal is to give these simple baseline methods to better show the effectiveness of prompt methods in a comprehensive comparison.
3. Indeed, we never claim that the single Null Prompt is novel or our contribution. Instead, our new strategy Multi-Null Prompt is to insert multiple [MASK]s at both start and end positions, which is different from Logan et al. Besides, we directly use embedding cosine distance as a metric to select similar label names. As for KPT, we respectfully do not agree that it undercuts the value of our paper. These two works focus on different scenarios, and the motivations are also from different views. As for KPT, it works on incorporating external knowledge into prompt tuning, while we are focusing on the zero-shot setting with only the Multi-Null Prompt [MASK]s, which is much simpler with no requirement of human knowledge or unlabeled data. Also, our goal is to give a comprehensive empirical study together with our simple strategy. Both works have their own values in different ways.
4. Our significant finding is multiple [MASK] tokens at both start and end positions can be surprisingly effective instead of the Null Prompt method that we utilized as a baseline method. Besides, the simple embedding cosine similarity-based label words extension is also effective. As we present, the knowledge-based methods perform better than our Multi-Null Prompt, but the simpleness and ease to use of our method greatly “simplifies the process of defining a prompt for new tasks for future researchers”, as you comment. We’d like to respectfully emphasize that the performance is only one view when considering the value for one work. The cost of the knowledge-based method is another view that needs to be considered. We agree that it is interesting to test whether our multiple [MASK]s improves with manual prompts. Therefore, we conduct experiments on the KPT method with our Multi-Null Prompt for text classifications, and the results show that this combination further improves the zero-shot performances. For example, the IMDB classification accuracy can be $93.58$ (our + KPT), compared with $86.59$ with Multi-Null Prompt (our), $91.50$ with KPT, where the accuracy is further improved. This clearly demonstrates that Multi-Null Prompt is effective and complementary with traditional manual prompt methods. We put this experimental study in Appendix A.2 in the updated version.
5. Thanks for your feedback on these studies, and let us clarify our motivations for these studies.  
   * For random positions. Start or end positions are the default and most straightforward positions to be tested, but we are curious about the impact of different positions since our method has no constraint of the [MASK] positions, and we aim to give a full study. It is interesting that the performance also gets improved, and better performance is achieved with more [MASK]s (though worse than start and end positions). This demonstrates that even randomly placing the [MASK]s also provides valuable information to benefit the text classification.
   * For other words beyond [MASK]. Though we mention that the MLM task replaces tokens with [MASK] or random tokens, the substitution probabilities of [MASK] and random tokens are largely different ($80$% vs. $10$%) during pre-training. Therefore, it is still very surprising to see the strong performance with other tokens compared with [MASK]. Besides, before conducting this experiment or before speculating the reasons, we normally/instinctively feel they should be much different. With the surprising results, we give our speculation, and we hope this can inspire researchers to think about the reasons beyond our speculation deeply.  
   * For different ensemble methods. We agree that mean pooling is the direct and natural ensemble method. However, similar to that in the general pretraining/finetuning scenario, there is no clear conclusion on which [CLS] or mean-pooling of the hidden states is better for prediction. Therefore, we conduct the different ensemble methods to test these choices. In a short summary, these studies aim to provide a comprehensive investigation and show a complete picture of our Multi-Null Prompt, and we also give some interesting and surprising results together with our analysis.


 Thanks for your insightful comments and reviews. We provide the following responses to your concerns. 
> It's unclear why / with what insight from the experiments the new strategy of Multi Null Prompt was proposed.

A. Sorry for not clearly presented motivation. Indeed, when we first notice that Null Prompt is effective from Logan IV et al. (2021), we are thinking of whether there are simple extended methods. One insight is from NLP tasks (e.g., multi-task training, multi-lingual training), it is usually effective to add a special token (to indicate the task or the language) during training, and empirically more distinct special tokens are more effective. Since the special [MASK] token is useful, we hence take the simplest way, that is to duplicate the [MASK] tokens, which is our Multi-Null Prompt strategy. Surprisingly, it works pretty effectively for text classification.  

> In Table 1, it seems that only some of the results have error bars?

A: Sorry for the unclear presentation in Table 1. The error bar stands for the variance for multiple more correlated label words experiments (the * in Table 1). That is, we conduct experiments with $30$, $60$, and $90$ correlated label words for * experiments, and then we report the average result and error bar in Table 1. We have revised the illustration in Table 1 in the updated version.

> In Table 3, it's unclear how the alternative words for MASK were chosen. Are they random words? 

A: The replaced tokens in Table 3 are selected randomly, without any specific rules. The reason is that, during pretraining, MLM task replaces tokens with [MASK] or random tokens with largely different substitution probabilities ($80$% vs. $10$%), we are curious about the random tokens as Null Prompt, and we normally/instinctively feel they should be much different. Surprisingly, when we conduct this experiment, it is interesting to see the strong performance with other tokens compared with [MASK]. Therefore, we also put this surprising study in the paper, and we hope to inspire researchers to think about the reasons deeply.

> Perhaps it would help to show how much it improves initially from zero additional words.

A: Thanks for your suggestions. We replot Figure 2 from zero additional words, and the curve is updated in the new version.  

> Multi Null Prompt cannot be used with some types of tasks that need multi MASK positions. Is there a proposal for how to deal with this limitation?

A: We are also thinking of possible proposals. Currently, we think one possible solution is to use multiple Multi-Mask Prompts. That is, we use multiple [MASK]s for each [MASK] position to formulate a multiple Multi-Mask prompt, which can be regarded as a hierarchical solution.  

> why is zero shot referred to as a more general setting than few shot? It seems the other way around.

A: Sorry for the description that zero-shot is more general. We would like to mention that zero-shot is more challenging than few-shot. We revised it in the updated version.

> In Fig 1, interestingly after about 3 MASK tokens, the performance tends to go down for Multi Null prompt. Any intuition for this?

A: Yes, it is interesting that 3 [MASK] tokens perform best. We are not so sure about that but let us try to give some hypothesis. When [MASK] tokens increase from 0-3, these tokens may be represented by different summarized information about the sequence [X]. When more and more [MASK]s involved, there may exist much redundant knowledge/information in these tokens so to decrease the performances. We will try to understand this in the future. 

> In Table 4, why does ALBERT have such a different behavior on IMDB?

A: It is interesting to see the result of ALBERT in IMDB differently from other tasks. Currently, we cannot give a perfect analysis of this observation and we just give an honest report. We may study this deeper later.

> In Fig 2, what does number 1, number 2 number 3 mean?

A: Sorry for the confusion of the notations of number 1/2/3 in Figure 2. The `number’ means the number of [MASK] tokens inserted at the start and end positions; we have revised it to be 1/2/3/4 [MASK]s in the new version. 

We hope the above answers can be helpful. 


 Thanks for your great comments and reviews. Below are our responses towards your concerns and questions. 
> The breadth of the contributions of the paper are questionable if you account for the fact that zero-shot has been explored as a baseline in [2] (Line 1, 2 of Table 1) and the efficacy of NULL prompting has been pointed out in [3].

A: Thanks for the related works. Indeed, we have cited [2] and [3] in Section 3.1, and their prompt methods are treated as compared baselines. Therefore, we do not claim the contribution of the Null Prompt strategy and the zero-shot exploration. Instead, the comprehensive study, simple Multi-Null Prompt strategy, coarse-to-fine studies, and the observed limitations are the contributions.
> Missing important comparisons

A: Thanks for your suggested important baselines. We conduct the two experiments accordingly. 

>> NULL prompting based methods benefit greatly from using the expanded vocabulary set. But the other methods being compared to are not evaluated with this expanded vocabulary set. I suspect that the other methods would benefit from this too. 


A: We put the (1) manual prompt with expanded vocabulary and the (2) null prompt with an expanded vocabulary. The results for the null prompt with expanded vocabulary are reported in Table 1 with Null*, and the results for the manual prompt with expanded vocabulary are as follows. For both Manual Prompt# and Manual Prompt (Prior), the average performances on the four classifications are improved by more than $3$ points. The detailed numbers are in the below table. From the table, we can find this is consistent with Null Prompt and Multi-Null Prompt strategies, as shown in the paper.

Manual Prompt with Expanded Vocabulary

Method | IMDB | Amazon | AG News | DBPedia | Average
---| ---| ---| ---| ---| ---
Manual Prompt# |  $74.06_{13.04}$ |  $75.54_{11.77}$ |  $67.65_{5.77}$ |  $56.75_{19.23}$ |  $68.50$
Manual Prompt#* |  $74.97_{9.57}$ |  $76.08_{11.83}$ |  $79.43_{2.29}$ |  $65.7_{14.95}$ |  $74.05$
Manual Prompt (Prior) |   $88.98_{4.12}$ |  $82.29_{11.08}$ |  $66.42_{9.58}$ | $69.25_{15.95}$  |  $76.74$
Manual Prompt (Prior)* |  $87.85_{5.93}$ |  $89.29_{5.09}$|  $72.12_{5.56}$ |  $69.11_{9.98}$ |  $79.59$

>> Multi-NULL seems to benefit from having multiple [MASK] tokens. Manual prompts with multiple mask tokens should be explored. 

A: According to your suggestion, we conduct experiments of the manual prompt with our Multi-Null Prompt on text classifications, and the result shows that this combination further improves the zero-shot performances. For example, the IMDB classification accuracy can be $94.00$, compared with $86.59$ with Multi-Null Prompt, the accuracy is significantly improved. This clearly demonstrates that Multi-Null Prompt is effective and complementary with traditional manual prompt methods. We put this experimental study in Appendix A.2 in the updated version.  

> How were the tokens in Table 3 that replaced [MASK] chosen ? It is curious to see that all the mask words chosen lead to better-than-random performance, are there any words that this fails on - ie - worse than random ?

A: The tokens in Table 3 are selected in a random way. We do not set any specific rules to select the tokens, except the ‘.’ token that fails. According to your question, we randomly select more (50) tokens to replace [MASK] and test the performance. Each result is also good compared to [MASK], the average, max, min, and standard deviation of the results are $77.27$, $81.17$, $65.68$, and $3.47$, which are comparable to the reported numbers in Table 3. We give speculation in Section 4.2 that it relates to the MLM pre-training task, but we are also interested in deeper analysis to explain it.

> Table 1 - why does *-NULL* methods have error bars but *-NULL method don’t ? Is the correlated label set dynamic ? If the correlated label set is dynamic, how is the set dynamically chosen ?

A: Sorry for the unclear illustration in Table 1. The 'Null*’ stands for multiple more correlated label words experiments. That is, we conduct experiments with $30$, $60$, and $90$ number of correlated label words for * experiments separately, and then we report the average result and error bar in Table 1. For the selection of the correlated words, we have introduced in Section 3.2 that they are based on the cosine distance similarity to the label name. In comparison, `Null’ stands for the original 1 label word experiment. We have revised the illustration in Table 1 in the updated version.

> Which tasks are you evaluating against in Figures 1 and 2 ? 

A: Sorry for the missed introduction of the tasks in Figure 1 and 2. They are on all four text classification datasets (IMDB, Amazon, AG News, DBPedia), and the average results are shown in the Figures. We have revised it in the paper. 

Thanks again, and we hope our answers can help address your concerns about the missing baselines and experimental details.  



 Thanks for your valuable comments and reviews. We provide the following responses to your concerns.
> The variants of the [mask] token for Section 4.2 seem to have only limited values to the work.

A: For variants of [MASK] tokens. Let us clarify the motivation of this study. During pretraining, the MLM task replaces tokens with not only [MASK] with $80$% substitution probability but also other random tokens with $10$% probability. Therefore, we are curious about the random tokens like [MASK] token as `Null’ Prompt, and we normally/instinctively feel their performances should be much different. However, we surprisingly find that other random tokens also achieve strong performance compared with [MASK]. Hence, we put this surprising study, and we hope to inspire a deeper analysis of this observation.
> The multiple null prompt may seem sort of simple technique and causes the concern whether the paper seems to meet the standard of novelty.

A: As for the novelty, we know that a simple technique sometimes causes concern, but it has two-side effects depending on multiple factors (e.g., result, efficiency) to assert the simple technique. For our simple Multi-Null Prompt, it significantly improves the performance of zero-shot text classifications (by more than $5$% accuracy), which is a nontrivial improvement. Therefore, a quite simple but effective method is widely accepted in the community. There are lots of examples, for example, dropout [1], cutoff [2], SimCSE[3]. Besides, we want to respectfully mention that the goal of this paper is to give a thorough empirical study to test the zero-shot capabilities of PLMs. The potential and limitations of the BERT PLMs are our focus instead of the technical novelty. But we will further improve in the future.  

> For the subword token as from Schick & Sch¨utze (2021a), can we combine the multiple null prompt from that paper w/ the multiple null prompt from current work? Please let me know -- suppose the authors know such results.

A: We are also interested in this combination in zero-shot learning. Currently, we think of one possible combination is to use multiple Multi-Mask Prompt. That is, we use multiple [MASK]s for each [MASK] position to formulate a multiple Multi-Mask prompt, which can be regarded as a hierarchical solution. We use the method from Schick & Schutze (2021a) under zero-shot text classification (IMDB), and the accuracy is $61.03$. After combing with our Mutil-Null Prompt, the accuracy can be improved to be $63.79$ (IMDB).  

We hope the above responses can help eliminate your concerns.

References:

[1] Srivastava, Nitish, et al. "Dropout: a simple way to prevent neural networks from overfitting." The journal of machine learning research 15.1 (2014): 1929-1958.

[2] Shen, Dinghan, et al. "A simple but tough-to-beat data augmentation approach for natural language understanding and generation." arXiv preprint arXiv:2009.13818 (2020).

[3] Gao, Tianyu, Xingcheng Yao, and Danqi Chen. "SimCSE: Simple Contrastive Learning of Sentence Embeddings." arXiv preprint arXiv:2104.08821 (2021).


 And for the minor issues:
1.	Thanks for your great comment and suggestion! We give examples of different prompt strategies in Appendix A.3 in the updated version. 
2.	We will conduct experiments according to your suggestions, but due to time limitations, we are not sure whether we can put the results in time. We will put it forward.  


Hope our responses can help address your concerns. 

References: 

[1] True Few-Shot Learning with Language Models, https://arxiv.org/pdf/2105.11447.pdf

[2] MULTITASK PROMPTED TRAINING ENABLES ZERO-SHOT TASK GENERALIZATION, https://arxiv.org/abs/2110.08207


### What is the Problem / Question?
How effective are pre-trained language models (PLMs) in zero-shot settings across diverse prompting strategies and tasks?

### Why is it impactful?
Recently, there have been a battery of studies examining the extent to which PLMs can serve as few- or zero-shot learners, finding that they can perform surprisingly well given minimal to no fine-tuning data simply by framing problems in LM settings via cloze-filling or prompting strategies. Gaining a better understanding of the utility of LMs in this setting, and how to best exploit their abilities, could therefore be very impactful, provided the insight gained offers sufficient inductive value.

### Why is it hard? Why have previous approaches failed?
Prompting is a relatively recent development in the study of PLMs, and while there have been studies examining various aspects of prompting (e.g., https://arxiv.org/pdf/2105.11447.pdf), comprehensive studies of the effect are limited given its novelty.

### How do the authors solve / answer it?
The authors profile 3 different BERT models over 20 different datasets using several prompting strategies. They perform several targeted analyses to better probe some of their findings (such as delving more deeply into the multi-null strategy) and also offer commentary on several overall takeaways. 

### How do the authors validate their solution / What is the answer to the question?
Ultimately the authors find several properties:
  1) Using the RoBERTa model as a base, across datasets the authors find that prior reported results for the power of prompt based zero-shot systems are very sensitive to the choice of prompts used, and that using other prompts may yield dramatically worse performance. This is consistent with prior literature, such as https://arxiv.org/pdf/2105.11447.pdf, which is not cited in this paper but likely should be.
  2) Using a variety of somewhat odd strategies in which prompts are not used but instead the representation fed to the MLM head is obtained through other means (either the CLS token, the SEP token, or mean pooling), performance drops dramatically vs. any prompting methods. This is not at all surprising, as those representations are never fed to the MLM head during PT, so this usage method is effectively untrained.
  3) Using a simple, unrefined prompt method of "[X] [MASK]", with the answer inferred from the [MASK] token is surprisingly effective, often matching the unspecialized manual prompts. However, as the authors note, this strategy overall is not a new strategy; [Logan et al.](https://arxiv.org/abs/2106.13353) explore it previously, so the impact of this contribution is limited. That said, in what I think is a new finding, it can be further improved by using multiple mask tokens at different prositions in the sentence and ensembling them together. This is a nice finding, as it simplifies the process of defining a prompt for new tasks for future researchers, though the fact that methods like KPT still take first undercuts its value somewhat.
  4) The authors further profile other methods which use unlabeled data or external knowledge to refine their learners, and find that these robustly outperform all simple prompt-based strategies. The authors further adapt some of the ideas from these other methods to their Null-prompt method, finding that using additional label words in their prompt resolution step improves performance significantly. Though such strategies have been previously explored (e.g., in the knowledge-based methods they cite), I'd argue this is their most significant finding; that by taking a method like Null-prompt and expanding the labelspace via synonyms, they can significantly boost performance. However, this value is somewhat undercut by the fact that the existing knowledge based methods still significantly outperform null-prompt based settings. It would therefore be very interesting to try to port this discovery into other prompting strategies -- e.g., do manual prompts improve if multiple [MASK]s are included? What about the KPT method?
  5) The authors further analyze how different variations on the multi-null method offer benefits, finding that placing up to 3 [MASK]s at both the start and end of the sentence and mean-pooling the results improves performance by up to ~5%. Inserting the [MASK]s randomly into the text is also explored, though not unsurprisingly this greatly reduces performance from the traditional start or end of sentence methods. The authors analyze a few other niche cases here, such as whether other words beyond [MASK] can be used and how to aggregate the [MASK] tokens, but in general I find these analyses to be less compelling as the questions answered seem less impactful to me; it is unsurprising that using random tokens doesn't hurt performance too dramatically given the setup of the MLM task (a fact which the authors note), and I would only normally conider mean-pooling or majority vote to be natural ensemble methods in this context, and they both generally perform comparably.
  6) The authors also analyze prompting strategies on a battery of other kinds of natural language understanding tasks, finding that models do much worse on these than on the simpler text-classification tasks they've already explored. The authors argue this is due to the difficulty of identifying high-quality label words, but I suspect a simpler explanation is more likely; that one is much less likely to find pre-training MLM examples in the PT corpora for these PLMs that resemble GLUE tasks than one is that resemble simple text classification tasks. The success of more recent approaches like T0pp also seem to contradict their intuition here, though of course T0pp had much more directed training to solve these sorts of tasks. Ultimately, as the authors do not find many instances of success on these more involved tasks, and there analysis of the reason behind the poor performance is limited to simple qualitative arguments, I'm not sure of the impact of this finding.
 ### Key Strengths
  1. I think prompting is a very exciting and promising area, and obtaining new transferrable insights into how it works would be very valuable.
  2. I think there are some interesting preliminary findings here, most notably (a) confirming that expanding the labelset words improves performance and (b) in finding that expanding the number of [MASK] tokens improves performance.

### Key Weaknesses
  1. Unfortunately, this field is fast moving and there are both other works that need to be cited (e.g., https://arxiv.org/pdf/2105.11447.pdf) and other works that have since come out that undercut the value of this work (e.g., https://arxiv.org/abs/2110.08207). I don't expect you to have profiled the latter -- I know it came out after the ICLR submission deadline. Nonetheless, its existence does undercut the impact of your work.
  2. Even ignoring T0pp, I'm not sure how much there is that I can take away from this work that will materially impact future research. Of the 6 major findings I identified in my summary above, most are either completely expected, undercut by other existing models which further improve performance, or already known. The direction that seems like it has the most potential of improving future research is the finding on multiple [MASK] tokens and the more general idea of ensembling different prompts / label-words together. I think a more focused analysis on that, and how it impacts other various prompting strategies, could have a higher potential of providing transferrable insight, but as it stands now I question the significance of this work.
  3. I think there are a few things you assess that seem not well-motivated, and those just clutter your work. In particular, the [CLS], [SEP], and mean-pooling rows in Table 1 and associated commentary on those methods doesn't seem to offer value here. I also question some of the ensemble strategies used for the [MASK] study, but as comparing mean-pooling and majority vote does seem well motivated, I'm not as confident you'd need to remove the other methods studied there.

### Minor issues
  1. I think the work would benefit from more clarity (perhaps some examples) of how each of the tested prompt strategies work (including previously published works)
  2. I'd love to see the knowledge systems also tested on GLUE or other NLU tasks.
 Unfortunately, as it stands while I think the work is well exectued, I question the impact it is likely to have on future research. Therefore, I'm not sure it is suitable for ICLR, and might recommend another venue, such as a workshop submission until further research is done that focuses on isolating and expanding these studies in areas that would offer transferrable insight for future research.

The paper has a full span of empirical results on BERT models trained by both methods of fine tuning and prompt. The study was on zero shot learning task, and the authors proposed two new methods -- multiple null prompts and the search for label words. The fine tuning was worse than prompt BERT models for zero shot setup. The new approaches from the paper have large F1 gains compared with single null prompt. The author extensively conducted empirical study on many setup, e.g. the number and combination of the null prompt, zero shot and few shot, the classification and GLUE task. Strength
- Proposed new method of search for label words, the multiple null prompts
- Abundant study
- Proposed multiple null prompt demonstrates very big enhancement

Maybe weakness
- The variants of the [mask] token for Section 4.2 seem to have only limited values to the work.
- The multiple null prompt may seem sort of simple technique and causes the concern whether the paper seems to meet the standard of novelty.
- For the subword token as from Schick & Sch¨utze (2021a), can we combine the multiple null prompt from that paper w/ the multiple null prompt from current work? Please let me know -- suppose the authors know such results.
 The result would clearly worth publishing to researchers working on prompt BERT model. The paper may be around the borderline for acceptance of the conference unfortunately, from the novelty concern that has been talked before.

This paper studies the zero-shot capabilities of the BERT family of models. The paper seeks to establish that simple variants of NULL-prompting can provide a competitive alternative to few-shot fine-tuning of BERT family models for text-classification. The paper analyzes these variants of NULL-prompting to establish principles behind design choices. The paper finally establishes limitations of NULL-prompting for zero-shot NLU tasks. Strengths
1. Extensive exploration of the different design choices that went into creating Multi-NULL\*. 
2. Exploration of the limitations of the zero-shot capabilities of BERT family models. It is good to see intuition confirmed about zero-shot NLU capabilities of BERT models
3. Paper is well written with the inclusion of substantive background information

Major Issues
1. The breadth of the contributions of the paper are questionable if you account for the fact that zero-shot has been explored as a baseline in [2] (Line 1, 2 of Table 1)  and the efficacy of NULL prompting has been pointed out in [3]. 
2. Missing important comparisons

      2.1 NULL prompting based methods benefit greatly from using the expanded vocabulary set. But the other methods being compared to are not evaluated with this expanded vocabulary set. I suspect that the other methods would benefit from this too. This is an important missing baseline because it would affect the paper’s conclusions (about methods requiring no human input and auxiliary data) if any of the representation aggregation approaches with the expanded vocabulary set outperform \*-NULL\* approaches

      2.2 Multi-NULL seems to benefit from having multiple [MASK] tokens. Manual prompts with multiple mask tokens should be explored. This baseline, if it performs as well as Multi-NULL would be a more principled approach.

Minor Issues
 1. Some important details are missing. Please see the questions below

Questions for Authors
 1. How were the tokens in Table 3 that replaced [MASK] chosen ? It is curious to see that all the mask words chosen lead to better-than-random performance, are there any words that this fails on - ie - worse than random ?
 2. Table 1 - why does \*-NULL\* methods have error bars but \*-NULL method don’t ? Is the correlated label set dynamic ? If the correlated label set is dynamic, how is the set dynamically chosen ?
 3. Which tasks are you evaluating against in Figures 1 and 2 ?  Would be easier to contextualize the results knowing the task. If this is an individual task, do the trends hold across other tasks ?


Relevant Related Work
1. Zero-shot text classification with generative language models [https://arxiv.org/pdf/1912.10165.pdf]
2. Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference [https://arxiv.org/pdf/2001.07676.pdf]
3. Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models [https://arxiv.org/pdf/2106.13353.pdf]


 Whilst the paper performs extensive experimentation to establish the limits of zero-shot capabilities of BERT family models, the comparisons made are questionable and some experimental details are missing. The novelty/significance of the paper’s premise is questionable when situated in the context of past work.
I am open to updating my score if my concerns about the missing baselines and experimental details are addressed.


Meta-review: The paper focuses on zero-shot capability of BERT-like models. The key contribution boils down to a novel prompting techniques that effectively ensembles predictions made for [MASK] tokens inserted at different places.

Reviewers B5Rv and 9k3X voted for rejecting mostly on the grounds that the contribution is not significant enough for ICLR. In particular, there are already existing works show that null prompting works, and other works that suggest that using multiple prompts works. While these insights have not been combined before, it is to some extend incrementally.

On the positive side, the multi-null prompting strategy is a genuinely useful tool. I think it is likely to find applications in different NLP applications as an effective way to generate ensemble predictions. The paper has also many carefully carried out experiments that will likely help guide future efforts in designing effective prompting strategies.

On the whole, I am recommending rejection. I know this is a disappointing result. Thank you for your submission, and I hope the reviews will help improve your paper.

Decision: Reject