Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents a novel method for zero-shot inference with masked language models (MLMs) such as BERT and RoBERTa for classification problems. The key idea is to use the encoders of the MLMs themselves to find relevant words for a given label name. For example, if the target label is "SPORTS", then the proposed method will first find the words by searching k-nearest neighbors with embedding distance. The embeddings are induced by the MLM itself. Finally, they aggregate the logits of these k words (for filling the masked positions). The authors argue that this new method (named NPPrompt) is a fully zero-shot classification method because it does not need any other information about labels. The empirical comparisons show that the proposed method has a comparable performance with other zero-shot methods (that needs some additional info such as KB or the unlabeled data). 


strength_and_weaknesses: ## Strengths

- The proposed NPPrompt method is simple and effective. It only requires the label strings for doing zero-shot classification. 
- The empirical results show that it is on the same level as other methods that need more information about the labels.

## Weakness 

- The proposed method is pretty limited to the classification tasks, particularly for news classification, where the label names are informative and have many semantic relevant words in the vocabulary. It is also limited when the label names are multi-word expressions or phrases. It does not support the cases where the label names are not informative enough and additional label description is needed. 

- The title and narrative can be misleading and an overclaim. As mentioned before, it is limited to masked LMs and the advantage is mainly for tasks like news classification. 

Suggestion:

- I suggest the authors extend the proposed method to *multi-label classification* settings where an example can have multiple acceptable news. 
- A major limitation is the multi-word expression and out-of-vocabluary label names.  How do you generalize the method to support that?
- Also, I suggest the authors try some multiple-choice QA problems.
clarity,_quality,_novelty_and_reproducibility: - The paper is mostly clear and the novelty is incremental. I think the reproducibility should be okay since the method is pretty naive. 

- Can you show the kNN results for sentiment analysis tasks and NLI tasks? I don't find them in the paper but they are quite important for qualitative analysis. 
summary_of_the_review: Overall, I think the paper presents a novel zero-shot classification paper but the limitations are there and the performance is not that significant. I think if the authors can extend the paper with my suggestions in the above section, the paper will be more suitable to the ICLR community. Otherwise, it will have much less impact to the field.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies how to effectively transfer pretrained language models to natural language understanding (NLU) tasks in a zero-shot manner. The proposed method, NPPrompt, does not require any labeled sample or rely on humans to construct prompt label words. NPPrompt generates the label words by searching the related words from the initial word embedding of the pre-trained language model. Then, it aggregates logits from the label words and predicts the category with the largest score. Experiments show that the proposed method is effective and outperforms strong baselines with a large margin.
strength_and_weaknesses: Strength
1. Proposed method is simple, intuitive and effective.
2. Conduct extensive experiments on a wide range of NLU tasks.

Weaknesses
1. The proposed method is not very novel. Generalizing LMs to NLU tasks by checking the vocabulary distribution over the masked token is a common practice. The method which leverages related words to label category does not look very novel to me.
clarity,_quality,_novelty_and_reproducibility: This paper is well-written. The work is original, but the proposed method is kind of similar to many existing works, which makes it less novel.
summary_of_the_review: This paper proposed NPPrompt, which aims to enable LMs' zero-shot ability to NLU tasks without requiring any labeled sample or relying on humans to construct prompt label words. Results show that NNPrompt outperforms strong baselines with a large margin. However, the overall novelty of this paper is limited.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a new language model named non parametric prompting PLM for natural language understanding specially for zero-shot learning. It is an important topic because these days many word-class associations are being produced by end users and previous models heavily depend on unlabeled data and human effort.  The authors showed that the proposed method outperforms state-of-the-art in terms of text classification accuracy and GLUE benchmarks on four different datasets including AG news, DBPedia, IMDB. and Amazon.
strength_and_weaknesses: The authors put significant effort on proving effectiveness of their method in a variety of NLP tasks. However, I wanted to see significant test results to make sure that the improvements are not random. 
clarity,_quality,_novelty_and_reproducibility: The paper was well written and easy to follow. I would require authors to add the github link for the code. 
summary_of_the_review: Overall, zero shot learning is an interesting topic in natural language processing as so many new categories and topics are being produced on the web. The authors proposed a simple and easy to implement method for pre trained language models to minimize human effort in terms of labeling and building training data. Overall I am satisfied with the current draft of the paper and request to move forward with discussion.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes an approach to zero-shot text classification. The idea is to only take the label name as supervision, to extend the label set to similar words using embeddings and to compute the probability of the different labels using MLM-based pretrained models. The method is evaluated empirically and shown to outperform baselines on topic and sentiment classification datasets.

It is nice to see that pretrained LMs can perform zero-shot classification only using the label identity. The proposed method for extending the label set is interesting.

However, this method works in very limited settings -- only for classification and when the class label is unambiguous. It is already known in this setting that the label identity is enough to achieve non-trivial performance. While the method improves the performance even further, it is unlikely that the particular method will have any practical use as it is much simpler to either label a few examples or write a short task description (e.g., prompt) rather than verify that the task is one in which the label identity is sufficient.
justification_for_why_not_higher_score: The novelty is rather limited -- extending the set of label words. The method is unlikely to be of practical importance as highlighted in the summary. Cherrypicking datasets on which the method will work leaves open the question of the limitation of the method.
Decision: Reject