Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper describes a dataset consisting of ~14k multiple-choice questions drawn from many different fields across the humanities and science as well as professional disciplines such as law and medicine. It presents results for GPT-3 models (LMs trained on text corpora with document context) of different scales, as well as for the UnifiedQA model (seq2seq model trained on various QA datasets). Performance of these models is well below their performance on other benchmarks: not above chance for the smaller GPT-3 models, and under 50% average accuracy for the best models.

The paper has a lot of good features: it’s obviously great to have a broad, challenging, large-scale dataset that aims to separate human from model performance. The lack of fine-tuning is a plus, as is having an objective evaluation procedure via multiple-choice answers. The calibration results for GPT-3 are very interesting.

However, I wonder about the value of a test that covers so many specialized areas of knowledge. A model that achieved expert performance (90% is suggested) across the board would clearly be superhuman, since no one is an expert in all these areas. Even a model that achieved expert performance in a few areas would probably be doing better than the average person. So instead of actually separating model performance from human performance, this benchmark might amount to just moving the goalposts about what constitutes human performance. To calibrate, it would have been helpful (albeit expensive, I realize) to see human scores from both experts and non-experts on this data.

Another problem is that the benchmark is likely to reward exposure to specialized domain data. Some tasks may benefit from this more than others, depending on the extent to which they are already represented in general-domain training corpora, and the extent to which they reward rote learning as opposed to true generalization. So it’s possible that the benchmark can be gamed by discovering these ‘easy’ tasks and doing aggressive data mining. The authors provide some evidence that this kind of approach isn’t likely to be fruitful for law, but that’s only one domain, and their experiment isn’t strictly comparable with, for instance, retraining GPT-3 on a corpus that includes many more legal documents. Ideally the paper would have included more evidence about the relative difficulty of tasks, linked to the amount of relevant training data for the models tested. Another option would have been to include only ‘hard’ tasks. But that begs the question about the value of using specialized tasks in the first place, as opposed to just training on general text and (more)  assiduously testing for world knowledge that’s unlikely to be explicitly represented (to borrow an example from the T5 paper: can a tuba fit into a backpack?).

Detailed comments and questions:
- Figure 2 runs up against the previous text.
- It would be good to merge UnifiedQA results (from figure 9) into figure 6.
- Is the plan to release this dataset publicly? There is no mention of this in the paper.
- To ensure a fair comparison, it would be interesting to run GPT-3 in the same text-generation mode as UnifiedQA.
review_title: Unsure if we should be testing for expert-level performance on 57 complex tasks.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: In this paper, the authors propose a new test to measure a text model’s multitask accuracy based on 57 different tasks in Humanities, Social Science and STEM subjects. The authors experiment with the GPT-3 models of various sizes and UnifiedQA, and the results suggest that the size of the model may be one of the important factors in achieving higher performance on all the tasks.

This is an interesting work that tries to tackle a very important problem of how successful are large language models across multiple tasks. It would help the paper to have a more thorough discussion of the results. Based on the reported results, larger GPT-3 models perform better but a smaller UnifiedQA model outperforms GPT-2 by a substantial margin, why is this? 

In addition, it would be interesting to see a comparison of GPT to BERT and XLNet. Such a comparison would first of all show if the idea generalizes to other types of language models. Second, it would emphasize the advantages/disadvantages of autoencoder-based vs autoregressive models and could potentially provide additional insights on how important these differences are for different tasks.

Given the authors provide additional discussion, I would like to see this paper and the associated dataset to be presented at the conference.
review_title: Interesting work on multitask performance of language models

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:
 
The paper proposes a benchmark for NLP models. The purpose of this test is to measure the model's knowledge in 57 topics covered by approx 15000 tasks in total, each formulated as a closed-form question in zero-shot and few-shot settings. Most of the tasks were taken from different human examination sets. Then, the paper provides results of experiments with the latest (GPT-3 and T5 based) models along with some quantitative and qualitative observations.


Pros:
+ Such a dataset of questions may be useful in the future if it will be published.
+ The authors show that in the case of GPT-3, as with earlier similar models, the model's confidence is not a good estimate of the actual probability of the prediction is correct.

Cons and questions:
- The paper compares only models of two different architectures, while the proposed format for comparing the probabilities of 4 tokens allows on to test even models of the BERT family.
- I believe it's important to understand the human level in such benchmarks. Moreover, it would be interesting to see several human baselines with different levels of education.
- It is known that the ability to answer zero-shot/few-shot questions depends on the size of the model, but the inability to answer in the zero-shot format does not necessarily mean a lack of necessary knowledge in the model. Thus, the proposed approach is biased towards larger models that are just better able to work in zero-shot mode.
- Despite the fact that the UnifiedQA model is superior in quality to GPT-3, most of the results are devoted to the GPT-3 model.
- The uneven success of models for different topics can possible be explained by several objective reasons (the average length of a topic question in tokens, the average frequency of topic question tokens in the corpus, the share of topic documents in the training sample, and so on). At least some of them can be checked within the paper.
- It's unclear if the authors are going to publish the questionnaire.

Overall, I vote for rejecting. I like the idea of a comprehensive questionnaire as an NLP benchmark but besides the proposed set of questions the paper includes only the results for two different architectures and some debatable hypotheses on the reasons for such results.

UPDATE:
Authors, thanks for your updates.
Some of cons are gone, and the paper is better now, but my main concern stays: it's unclear if the results are about the problem solving ability or the zero-shot learning ability. Thus, I corrected my score to from 4 to 5.
review_title: Recommendation to Reject

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper focuses on coming up with 57 different tasks and measure the performance of these large scale transformer models such as GPT3 on these different tasks. The main claims of this paper are to demonstrate these large-scale models still struggle to use the knowledge it has learned during the pretraining phase and these models struggle to on calculation-intensive tasks. Further one of the more important contributions of this work includes the massive multi-task dataset that comprises 57 different subjects.

1. This study shows how far off these language models are when compared to how humans use knowledge and commonsense reasoning to solve tasks. Along with that the collecting a comprehensive multi-task dataset that covers the depth and breadth of the multiple subjects would help the community understand these models better. 
2.  One important comparison that would have definitely helped is trying to understand the gaps in knowledge between a GPT-3 to Unified QA. The difference in performance across the tasks seems high especially comparing an 11B to 175B.

Questions:
1. How would someone reproduce these experiments considering that widespread access GPT-3 models are not available to the general research community? 
2. How was the level of difficulty of a subject measured? Was it already available when the datasets were being obtained?
3. Can you elaborate on the results from table 1 which shows that an 11B UnifiedQA model outperforms 175B GPT-3 model? Are these statistical significant?


===================== After reading the rebuttal ========
I thank the authors for providing further information and answering the question raised by the reviewers. Based on the responses and clarifying the issues I had, I have adjusted my score accordingly and improved it from a 5 to 6. 




review_title: This paper focuses on coming up with 57 different tasks and measure the performance of these large scale transformer models such as GPT3 on these different tasks.

Meta-review: This paper presents an extensive evaluation of two language models: GPT-3 and UnifiedQA on 57 tasks. The results demonstrate that these models are still far from expert-level accuracy and do not know when they are wrong.

I think this is an interesting paper that provides useful insights into the capability of large-scale language models. The authors also plan to release their dataset and have addressed some of the concerns from the reviewers to improve the paper during the rebuttal period.
Decision: Accept (Poster)