Venue: ICLR 2022 Track Blog

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This blog post discusses the advantages of framing NLP tasks classically treated as classification problems as auto-regressive generation tasks. The main part of the exposition is based on [1] which focuses on entity linking and entity retrieval. The presentation of the article is clear and well written. It is followed by a discussion about the more general advantages of treating NLP tasks as classification problems, and the connection with prompt-based learning. This area has seen a lot of recent development following the success of GPT-3, which has shown that it is possible to obtain interesting zero-shot performance on a lot of tasks by treating them as generation tasks with a carefully chosen prefix. The main issue of this blog post are the exposition and arguments developed in this last part, which are not always convincing, and lack nuances.

Pros:
- Clear and well written.

- Timely, knowledge intensive NLP and prompting to manipulate large generative models are two very active areas of research.

Cons:
- The view on generation versus classification developed in the last section is not balanced enough and doesn't reflect the issues of generation approaches. Tackling NLP tasks as generation problems makes a lot of sense in the context of GPT, where 1) a large and capable generative model is available and 2) the model is used in a zero-shot manner. This also brings closer the dream of having one model, able to generate text and to deal with a myriad of tasks just as well as more specialized models. But as far as I know there is still a performance gap on many tasks between BERT-like models and generative models. For instance T5 [2], a sequence to sequence model, is outperformed by versions of bi-directional transformers posterior to BERT trained with a masked language modeling objective such as Roberta on most of the tasks at an equivalent number of parameters, flops or inference time.

- I think this blog post could be improved by discussing the following point: how to decide if a certain task is good fit for a generative approach. The generative approach works well for entity linking, but what about other tasks? I believe there are specific reasons why it works well for entity linking, which might not transfer to other tasks.

[1] De Cao, N., Izacard, G., Riedel, S., & Petroni, F. (2020). Autoregressive entity retrieval. arXiv preprint arXiv:2010.00904.

[2] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu (2019). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. arXiv preprint arXiv:1910.10683

review_title: Review for "Classification or Generation? Understanding Paradigm Shift for Knowledge-Intensive Tasks"

Meta-review: We appreciated the choice of the topic and the clarity of the writing but we judged that the authors could have taken a step back from the paper and discussed some questions pointed out by the reviewers that give some perspective on the problem. 
Decision: Reject