Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: Fleet is a new benchmark and leaderboard that combines previous work on few-shot text classification. While there are no new tasks, the new benhmark adds value by combining complimentary lines of work in a well designed evaluation framework.

main_review: This paper presents a new benchmark and associated leaderboard that aggregates text classification tasks from three previous works on few-shot learning.

The tasks are chosen to cover a range of domain, task, and learning transfer. There is a detailed investigation of the trade-off between large test sets with few test episodes, and smaller test sets with many test episodes. The test setting is chosen to maximise precision of the benchmark given a moderate compute budget.

The paper presents a new prompt-based approach to few-shot learning that converts each of the classification tasks into a multi-choice QA task. The details of this conversion are not given in the paper and I would like to see them listed. Comparisons are also made to previous models that were applied to some subset of the tasks included in Fleet.

While this paper does not introduce any new tasks or evaluation methodologies, it does add value by combining several lines of complimentary work on few-shot learning into a single evaluation framework that has been well designed. The prompt based UniFew baseline also seems both simple and effective, although I would like to see more details of the prompt template design in the main paper.

limitations_and_societal_impact: The authors point out that Fleet is limited by it's restriction to text-classification tasks. It is clearly not a defnitive benchmark, but still valuable.

The paper hints at the design of templates for the UniFew baseline but does not provide enough details for replication.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors introduce FLEET a new benchmark for evaluating few-shot learning. It's focused on classification tasks in NLP. The features of this new benchmark are:
- evaluation of 4 different types of transfer
- textual labels for zero-shot classification
- experimentally determined sample size

In addition, they propose a new strong baseline called UniFew which is prompt-based model built on top of UnifiedQA. They show that this method is competitive with other few-shot systems. Finally, they release the code and framework for further extension.
main_review: Overall, this work is going to be useful for the few-shot learning community.

Positives:
- Coverage across 20 different classification tasks
- Evaluation of different transfer types like class transfer, domain transfer, etc.
- Incorporating learnings from other fields like computer vision.
- Proposal of a strong baseline which is evaluated on the benchmark and compared against other SOTA techniques.

Negatives:
- The benchmark only includes classification tasks and in English only. More content on how this can be expanded to multilingual settings and other task types would be useful.
- More details on how the benchmark will look and how it's been designed would have been helpful. Screenshots of the baselines and the metrics reported would be good.


Questions for the authors:
- Can existing benchmarks like GLUE or SuperGLUE be converted to few-shot benchmarks by creating few-shot versions of them? Wouldn't that be better since we will know both few-shot and fully supervised performance on the same set of tasks? It's a bit suboptimal for everyone to create a different set of tasks as a benchmark.

Minor comments:
- Include references about multilingual benchmarks: XTREME, XGLUE, XTREME-R
- Number of shots and classes: the numbers chosen here are a bit arbitrary and more concrete experiments could have been better.
- The list of tasks is in the Appendix. This is important and should have been included in the main paper

limitations_and_societal_impact: Yes they have with their short statement. They should have included considerations that go into selecting the "few" examples that the models fine-tune on. These need to be free of any bias.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors of this paper present a few-shot NLP benchmark called FLEET. FLEET consists of a group of NLP datasets and evaluates four transfer settings: class transfer, domain transfer, task transfer, and pre-train transfer. The authors also provide a simple yet effective prompt-based learning baseline along with the FLEET benchmark.
main_review: Strength:

1. A new few-shot benchmark with a variety of datasets and tasks. It provides separate evaluations for each transfer type and should be able to provide a comprehensive evaluation proxy for few-shot NLP frameworks.
2. Addresses several issues existing in previous few-shot NLP benchmarks, e.g. class imbalance and inconsistent problem settings.
3. A simple prompt-based learning framework with a decent performance on FLEET and other test sets that were used in previous work.

Weakness & Questions:

1. Currently, FLEET only contains one entity typing dataset (CoNLL). This dataset is only used for task & pre-train transfer. One could a) include more entity typing datasets b) and offer class transfer or domain transfer on the entity typing task.
limitations_and_societal_impact: The authors provide a section addressing the limitations in the provided benchmark.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes FLEET, a benchmark for few-shot learning in NLP. FLEET is comprised of different few-shot settings (new tasks, label sets, or domains at test time), different amount of training examples ("shots"),  and meta-learning strategies (w/ and w/o meta learning). The paper goes through the design decisions behind FLEET and detail its construction. Finally, they present a baseline based on UnifiedQA, wherein they reformulate downstream tasks as QA and then finetune the model on downstream tasks.
main_review: First, the benchmark contains a lot of different settings. On one hand, this is a positive as it allows many different types of research to be conducted with the data. On the other hand, it practically makes understanding, reporting, and using the benchmark a bit messy. For example, people will report very specific settings such as FLEET with {4-shot, with meta learning, task transfer}. More generally, there is often a tradeoff when constructing datasets of "do we focus on specific problems/settings we think are interesting" versus "do we build datasets that many people could use in different settings". This paper is a very extreme on trying to satisfy the second question: rather than trying to take a stand on what the community should work on, it tries to do a bit of everything. 

Second, a common criticism of GLUE and other benchmarks which applies to FLEET is that the benchmarks (1) simply repackage and resell other datasets as a new benchmark without adding substantial new value and (2) contain a relatively ad-hoc selection of individual datasets. FLEET is similar in that it sort of throws together a collection of tasks (sentiment, NLI, etc.) mainly guided by low-level reasons (e.g., the class imbalances) rather than higher-level ones (e.g., based on practical few-shot scenarios, different types of reasoning, tests of spurious correlations, etc.).

Third, a lower-level point is that I think few-shot benchmarks should contain multiple prompts / label descriptions for each dataset. There can be high variance across different prompts (Zhao et al. ICML 2021, Gao et al. ACL 2021, etc.) and so I think the reaction to this should be to report accuracy across multiple different prompts.     

Finally, the proposed method is extremely similar to this concurrent work https://arxiv.org/abs/2104.04670. This is not a negative or a positive, I am simply pointing this out as an FYI to the authors, as they should probably cite/discuss this in a future version of their draft.
limitations_and_societal_impact: Yes.
review_title: null

Meta-review: This paper introduces a family of multi-task benchmarks for few-shot learning in NLP. It appears to be built on existing data with relatively lightweight new methodological contributions, but reviewers agreed that it represents a useful tool and a clear improvement over current standard practice.

I'd urge the authors to add some discussion of the licences that apply to the datasets used. This issue came up only toward the end of reviewer discussion, and so won't influence our decision about the paper, but it's not currently clear whether the required data for the benchmark is guaranteed to remain legally available or whether private firms can use it.
Decision: Accept (Poster)