Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This paper propose a benchmark of few-shot text classification. Recently, large scale pre-trained language models such as GPT-3 as demonstrated strong few-shot capability. The authors, however, argued that AI models' few-shot capability should be evaluated in "real-world" settings, and build an evaluation benchmark of 11 sub tasks. The study show that SOTA few-shot model (GPT-3) still underperforms non-expert human on 8 of the sub tasks.
strengths: - strong baseline (GPT-3)
- human performance is included to set a clear goal.
- integration with popular opensource NLP toolkit (huggingface)


weaknesses: - lack of baseline models
- most tasks are existing datasets

In my opinion, GPT-3 is not widely available to the research community. Researchers in universities often do not have the infrastructure to run experiments at GPT-3 scale (175B parameters). I would suggest adding more baseline scores from other pre-trained transformers such as BERT, GPT-2, Roberta, etc, so that the benchmark is more accessible to the research community.

Given that the only experimental result is from a private API (GPT-3 from OpenAI) and most of the tasks are existing benchmark, I think this paper does not make enough contribution.
correctness: This benchmark has appropriate evaluation.
clarity: This paper is well written.
relation to prior work: Related works are properly discussed.
additional feedback: Include more accessible baselines such as GPT-2, RoBERTa would help reach a larger community.
review_title: Official Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper proposes a real-world few-shot text classification benchmark, where the classification task instructions are in natural language (for the system to process) and the tasks are of real-world value (e.g. some task are difficult even for non-expert humans). The authors constructed their RAFT dataset by selecting 11 datasets across different areas. The authors designed clear and straightforward experimental and evaluation settings for this dataset. In the experiments, a GPT-3 baseline is implemented and tested against human performances.
strengths: The most novel part of this dataset is including natural language instructions for the classification tasks. The authors presented several nice examples of such instructions, which are natural and reasonable in real-world scenarios.

As for the dataset itself, the data collecting procedure and the task settings are clear and sound.
weaknesses: The idea of including natural language task specification is interesting, but it seems that it is treated merely as a context in this paper. It is important to investigate these natural language instructions when constructing the dataset, since it is the most novel part of this dataset. For example, how different phrasing of the same classification task impact the performances are not investigated. Also, the instructions for different tasks in this dataset are completely separate. Given the same set of documents (e.g. semiconductor papers), we may ask the system to classify the institutions, or classify the papers by topics, etc. (otherwise it is not so meaningful to have natural language instructions). Another simple but interesting experiment to add is to test systems' performances on mismatched instruction/partially-labeled dataset, to investigate the true impact of task instructions (since all labels have natural language names, the performances might not be too bad).

From Table 2, it seems that the GPT-3 baseline already achieves reasonably good (or near-human) performances on 6 out of 11 tasks. Are these tasks too simple to be added to this dataset?


correctness: The dataset is constructed in a sound way, and the baseline experiments and evaluation are designed appropriately.
clarity: The paper is well-written and easy to follow.
relation to prior work: Previous related works are discussed.
additional feedback: Please see the weaknesses section.
review_title: Few-shot classification from real-world instructions, clear and straightforward setting

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: Authors propose a new few shot text classification dataset and benchmark, RAFT. Authors claim that the proposed dataset is more realistic and mimics exact real world text classification task. Authors evaluation on RAFT reveals that existing text classification techniques struggle with RAFT dataset. Authors have also put together a leader board to track progress on RAFT. 
strengths: - A few shot text classification dataset mimicking real world task.
- A carefully designed dataset with baseline result and leaderboard
weaknesses: - The main motivation for the dataset is not very clear as unlabelled test set in real world setting is easy to get.
- The zero shot baseline comparison is missing. 
- For a new "domain" users will have to get 50 instances labelled. So, the dataset doesn't server the purpose.
- Few shot in low resource setting could have made much sense.
- Relatively small dataset
correctness: Seems good.
clarity: The paper overall is very well written.
relation to prior work: Yes, discussed.
additional feedback: what does authors mean by "representative of real-world classification tasks." is not very clear in section 2.2.
review_title: A new few shot text classification benchmark

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: The paper proposes a novel text classification benchmark consisting of real-world datasets with 50 training examples each. The main contributions are:
- review of available tasks and the selection process itself,
- annotation of a few novel datasets, provision of meaningful labels where required for existing ones,
- provision of human baseline estimates and GPT-3 baseline,
- preparation of evaluation platform.

strengths: 1. The paper is easy to follow, and the problem of benchmarking few-shot classification systems is well-motivated. The benchmark differs from existing few-shot text classification benchmarks.
2. The selection process and evaluation setting reflect real-world use cases.
3. The evaluation process is mostly well-conceived and polished up, e.g., the authors presumably took into account that without a development set, submitters will have an incentive to optimize hyperparameters on the test set (and thus introduced a weekly evaluation).
weaknesses: 1. I do not expect papers that introduce benchmarks to place much effort on proposing the baselines. Nevertheless, the choice of GPT-3 as the only baseline in the classification task with 50 examples does not seem optimal to me. In particular, it is hard to assess how challenging are the proposed tasks given only GPT results – its performance might be anywhere between poor and excellent. The fact score is close to the human baseline does not help here because you relied on people without domain expertise (and thus, it is hard to tell what performance we should expect from experts in real-world application). I encourage authors to include an estimation of the score with the majority answer and some straightforward non-neural baseline. For example, XGBOOST can perform well with this number of examples on both short and long documents you tackle. 
2.  I wonder how much the selection of these particular 50 documents impacts the overall score on the task, i.e., what would be the variance with the repeated random sub-sampling validation procedure. If low, then your more straightforward evaluation procedure with fixed training examples is justified. Nevertheless, such analysis was not conducted, and thus, I am concerned that there will be a significant noise level while comparing different solutions. It is hard to assess the benchmark utility without a similar analysis conducted with GPT-3 or another model.
correctness: Newly-created datasets seem to be constructed in a sound way. Experiment design is probably correct. 
clarity: The paper is well-written and easy to follow.
relation to prior work: Differences are discussed in the introduction and Section 2.
additional feedback: I do not consider it a weakness, but the number of examples seems arbitrary at the moment – even though you refer to "authors’ experience with users of the classification tool." If you are interested in a few-shot evaluation and 50 is a limit, why not provide other train sets with, e.g., 5, 10, and 25 examples? I expect people to use the benchmark in this manner anyway. Moreover, the inclusion of Banking77 with a larger number of classes than training examples is controversial – it is largely a zero-shot scenario where different architectures are supposed to work well. 

Typo:
- There is a space before the dot in #144.
review_title: Important problem

Meta-review: This paper presents the RAFT which is a real-world few-shot text classification benchmark containing 11 sub tasks from different domains. This paper received borderline scores of (5,5,6,6). While all reviewers basically acknowledged that the dataset and the tasks are well-designed, the main concern was the lack of baselines other than GPT-3 (the result on GPT-3 itself is appreciated, though). Given the review, the authors updated the paper with the results on many other baseline models, which was appreciated and two reviewers agreed to raise the score from 5 to 6, although one of them is not reflected in the system. Thus all reviewers are now in the positive side.
Overall, there is not yet a particularly strong support for this paper, the AC considers it has no critical weakness and above the acceptance threshold.
Decision: Accept