Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The paper emphasizes existing dataset limitations that undermine claims of achieving broad human-level performance on NLU tasks. The main point is that previous works have been trained on a large corpus of data, which is not similar to the human learning process, thus comparing them would not be valid.

Based on this premise, the paper presents benchmark few-shot datasets for 6 NLU tasks and compares results of existing models on them.
strengths: - The subject is very well explained and important for the field.
- The paper is well-written.
- Six tasks are chosen from different aspects.
weaknesses: - In many datasets, there exist a separate test set. Are the results reported based on evaluation on those test sets or an average on randomly split sets? If the latter, please explain why not to use the already existing standard test sets.

- In Table 6, the validation report for the BERT model in CoNLL (and WikiANN) is the same for all values of K. In addition, the variance is 0 for 5 validation splits. Is there a specific reason for that, considering that other BERT-based approaches differ in values.

- In the first task, values less than 50 are reported for DeBERTa and some other models. Since this is a binary task, is there an explanation for that low accuracy?
correctness: Personally, I don't think there is a need to emphasize "we do not include a separate validation set for any tasks". As long as the dataset is providing a separate public/private test set, validation is a part of the training process and a strict validation split is rarely used in the latest approaches. Cross-validation is a very good substitute, which is decided by the ML engineer.
clarity: OK
relation to prior work: OK
additional feedback: Since NLP and NLU are not just about texts, I would recommend paying attention to tasks and benchmarks on other types of data that fall within the scope.
review_title: Very well researched, structured and presented study with strong impact and application

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper proposes a benchmark for few-shot learning in natural language understanding (NLU) called CLUES, taking the gap between human and machine learning into consideration. The motivation behind creating this benchmark is that the NLU models generally are trained/fine-tuned with large-scale task-specific data whereas the human can generalize the learned knowledge to these tasks with few examples. The main idea is also straightforward: sample a few-shot training set then ask human annotators to simulate the few-shot learning process using this dataset. Finally, the authors compare the machine and the human performance, and the results reveal that there exist large performance gaps between human and machine in the few-shot learning tasks regardless of fine-tuning strategies, model families, etc. 

The authors not only contribute a new few-shot learning benchmark for NLU involving human performance, but also provide the detailed comparison and preliminary analysis in terms of the performance gap.

strengths: 1. The motivation of the work is valid. Since the human and pre-trained language model both have general-domain "knowledge", it is unknown how they perform differently in the few-shot learning NLU tasks. 


2. The provided benchmark (CLUES) provides a fair setting to compare machine and human performance given a few training examples across diverse tasks. The authors also designed an annotation framework to measure the human performance on CLUES tasks and make the experiment reliable.


3. They conducted a large number of experiments considering the model families and fine-tuning strategies, and also analyzed the influence of different strategies on the few-shot learning ability of pre-train language models. It may enlighten future research.

weaknesses: 1. The comparison experiment is not complete. As mentioned in Section.5.1.(b), the authors did not design the prompt-based approaches for NER and MRC tasks. Though the performance gap is large between human and machine in these tasks, the experiments must be consistent enough.


2. One minor weakness of the motivation: Humans learned the knowledge in a multimodal way including visual clues and text information, but the pre-trained language only pre-trained on the text information. If we compare the few-shot learning ability, it may not 
be fair from this perspective.


3. The creation details of the datasets are not clear. See "Additional Feedback" parts.

correctness: The benchmark construction and the evaluation method look sound. But in line 202, the authors mentioned that:
> … the overall *S1* is computed as the average of *S1* of all the instances ... 

*S1* is a metric, not a numerical value. I suggest that using the following expression: 

> … the overall *S1* score is computed as the average of *S1* scores of all the instances ...

clarity: The paper is well written and organized except for:
1. Section.5, the best results in the tables should use **bold** font.
2. The symbols of S1 metric are not consistent. Should we use *S1* or $S1$ ?

relation to prior work: The discussion of related work is detailed with appropriate references of previous work.
additional feedback: I have some questions for the authors:

1. What is the sampling method of training and test data? Did you claim the seed of the random sampling method? Do you have a better way of splitting the data if we want to make the sampled data more challenging/discriminative.


2. Are the educational background and the first language of annotators the same?


3. For tuning the model in a task-specific way, some models need to add extra layers. It will increase the size of model parameters. Which type of model capacity did you report in Table 5? Original or Modified models?

There are some typos in the paper:
- Line 42: spans.^2. -> spans^2.
- Line 81: exiting -> existing
review_title: A benchmark for evaluating the few-shot learning capabilities of NLU models

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: Most of the existing benchmarks for NLU give models access to relatively large amounts of labelled data for training. This paper introduces a 
standardized benchmark and principles to evaluate few-shot NLU models. This benchmark has been designed for a relatively fair comparison between human and machine performance on three types of NLU tasks including sentence classification, sequence labelling and machine reading comprehension. Different pre-trained language models and adaptation methods have been evaluated on the proposed benchmark.

strengths: 1. This paper provides a standardized benchmark for evaluating different few-shot learning approaches.
2. Results for both humans and models with different adaption techniques are further provided. The proposed benchmark may facilitate progress in bridging the relatively large performance gap between humans and machines.
3. The design principles of the benchmark including validation, variance and task difficulty are relatively sound.

weaknesses: 1. For subsection 3.1 Task selection, I do not see the explicit connections between the design principles (Table 1) and the chosen tasks.  For example, which principle is considered when choosing CoNLL03 and WikiANN as the tasks. Why not other sequence labelling tasks?
2. For the related work section, the paragraph on few-shot learning in NLU can be extended a bit to give a more comprehensive review of recent efforts on few-shot learning and why such a standardized benchmark is needed. As mentioned by the other reviewer, I will also suggest the author provide more detailed comparisons with the existing few shot NLU benchmarks (FewGLUE [1], and FewCLUE [2]). Based on the existing submission, I can see the differences between CLUES and CROSSFIT.
3. The implementation of baselines is not publicly available.

[1] It’s not just size that matters: Small language models are also few-shot learner. NAACL 2021 
[2] FewCLUE: A Chinese Few-shot Learning Evaluation Benchmark. 2021

correctness: The construction of the benchmark is relatively sound given their goals. The experimental settings and baselines they established for their benchmark seem sound.
clarity: The paper is easy to read and follow.
relation to prior work: The related work section gives a brief introduction of few-shot learning in NLU, and an overview of existing NLU benchmarks.
additional feedback: No
review_title: A promising benchmark for few-shot NLU

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper proposes CLUES, a benchmark for evaluating the few-shot learning capabilities of NLU models. CLUES includes two sentence classification tasks (SST-2 and MNLI), two machine reading comprehension tasks (SQuADv2 and ReCoRD), and two sequence labeling tasks (CoNLL03 and WikiANN), where each task contains 10 to 30 samples for training. All the tasks are formulated as extractive QA problems and evaluated under a true few-shot setting (no extra validation set available). The authors applied multiple pre-trained language models, such as BERT, RoBERTa, T5, GPT3, with 3 different few-shot strategies (fine-tuning, prompt-based fine-tuning, and in-context learning). Experimental results show substantial gaps between the best model and human baseline in most of the tasks.
strengths: - The authors use a unified task formulation and propose an evaluation metric (S1) for evaluating all the tasks in CLUES.
- The authors have done a good job implementing multiple baselines with different language models and few-shot strategies.

weaknesses: - The authors should have made some direct comparisons with the existing few shot NLU benchmarks (FewGLUE [1], CROSSFIT [2], and FewCLUE [3]) to highlight the advantages of CLUES.
- There could be an error analysis to help readers understand the limitations of current SOTA few-shot models in the CLUES benchmark.
- The implementation of baselines is not publicly available.


[1] It’s not just size that matters: Small language models are also few-shot learner. NAACL 2021
[2] CROSSFIT : A Few-shot Learning Challenge for Cross-task Generalization in NLP. 2021
[3] FewCLUE: A Chinese Few-shot Learning Evaluation Benchmark. 2021
correctness: The data collection and evaluation methods are reasonable to me.
clarity: This paper is well written and easy to follow.
relation to prior work: As mentioned in weakness, authors could make a table and list all the few-shot NLU benchmarks and highlight the advantages of CLUES.
additional feedback: None
review_title: A Few-Shot Learning Benchmark for Natural Language Understanding.

Meta-review: This paper presents a benchmark dataset for few-shot learning based on the GLUE benchmark for natural langauge understanding. While there exist previous benchmark datasets, this work presents a new benchmark with significant novel contributions over the previous datasets. This would be a useful resource for NLP researchers.
Decision: Accept