Measuring Massive Multitask Language Understanding
Authors: Dan Hendrycks
Collin Burns
Steven Basart
Andy Zou
Mantas Mazeika
Dawn Song
Jacob Steinhardt
Venue: ICLR 2021 Poster (2021)
Keywords: multitask
few-shot

Abstract: We propose a new test to measure a text model's multitask accuracy. The test covers 57 tasks including elementary mathematics, US history, computer science, law, and more. To attain high accuracy on this test, models must possess extensive world knowledge and problem solving ability. We find that while most recent models have near random-chance accuracy, the very largest GPT-3 model improves over random chance by almost 20 percentage points on average. However, on every one of the 57 tasks, the best models still need substantial improvements before they can reach expert-level accuracy. Models also have lopsided performance and frequently do not know when they are wrong. Worse, they still have near-random accuracy on some socially important subjects such as morality and law. By comprehensively evaluating the breadth and depth of a model's academic and professional understanding, our test can be used to analyze models across many tasks and to identify important shortcomings.

Published as a conference paper at ICLR 2021
MEASURING MASSIVE MULTITASK LANGUAGE UNDERSTANDING
Dan Hendrycks UC Berkeley
Collin Burns Columbia University
Steven Basart UChicago
Andy Zou UC Berkeley
Mantas Mazeika
Dawn Song
Jacob Steinhardt UC Berkeley
UIUC
UC Berkeley
ABSTRACT
We propose new test to measure a text model's multitask accuracy. The test covers 57 tasks including elementary mathematics. US history, computer science, law, and more. To attain high accuracy on this test, models must possess extensive world knowledge and problem solving ability. We find that while most recent models have near random-chanee accuracy, the very largest GPT-3 model improves over random chance by almost 20 percentage points on average. However, on every one of the 57 tasks, the best models still need substantial improvements before they can reach expert-level accuracy. Models also have lopsided performance and frequently do not know when they are wrong. Worse, they still have near- random accuracy on some socially important subjects such as morality and law. By comprehensively evaluating the breadth and depth of model's academic and professional understanding, our test can be used to analyze models across many tasks and to identify important shortcomings.
INTRODUCTION
Natural Language Processing (NLP) models have achieved superhuman performance on a number of recently proposed benchmarks. However, these models are still well below human level performance for language understanding as whole, suggesting disconnect between our benchmarks and the actual capabilities of these models. The General Language Understanding Evaluation benchmark (GLUE) (Wang et al., 2018) was introduced in 2018 to evaluate performance on a wide range of NLPP tasks, and top models achieved superhuman performance within year. To address the shortcomings of GLUE, researchers designed the SuperGLUE benchmark with more difficult tasks (Wang et al., 2019). About year since the release of SuperGLUE, performance is again essentially human-level (Raffel et al., 2019). While these benchmarks evaluate linguistic skills more than overall language understanding, array of commonsense benchmarks have been proposed to measure basic reasoning and everyday knowledge (Zellers et al., 2019; Huang et al., 2019; Bisk et al., 2019). However, these recent benchmarks have similarly seen rapid progress (Khashabi et al., 2020). Overall, the human-level performance on these benchmarks suggests that they are not capturing important facets of language understanding.
Transformer models have driven this recent progress by pretraining on massive text corpora, including all of Wikipedia, thousands of books, and numerous websites. These models consequently see extensive information about specialized topics, most of which is not assessed by existing NLP benchmarks. consequently remains an open question just how capable current language models are at learning and applying knowledge from many domains.
To bridge the gap between the wide-ranging knowledge that models see during pretraining and the existing measures of success, we introduce new benchmark for assessing models across diverse set of subjects that humans learn. We design the benchmark to measure knowledge acquired during pretraining by evaluating models exclusively in zero-shot and few-shot settings. This makes the benchmark more challenging and more similar to how we evaluate humans. The benchmark covers 57 subjects across STEM, the humanities, the social sciences, and more. It ranges in difficulty from an elementary level to an advanced professional level, and tests both world knowledge and problem solving ability. Subjects range from traditional areas, such mathematics and history, more

---

Published as a conference paper at ICLR 2021
GPT-3 Few Shot Test Performance
Few Shot Prompt and Predicted Answer
90
Commonsense
The following are multiple choice questions
80
Linguistics
about high school mathematics.
Knowledge (Ours)
How many numbers are in the list 25, 26, 100?
Performance (%)
70
(A) 75 (B) 76 (C) 22 (D)
60
Answer: B
Compute/ i+ / + ........ 1 250 (A)-1 (B) (D)
50
40
Answer: A
4 daps 7 yaps, and yaps baps, how many daps equal 42 baps?
30
20
(A) 28 (B) 21 (C) 40 (D) 30
Small
Medium
Large
X-Large
Answer: C
Model Size
(a) An example of few-shot learning and inference us- ing GPT-3. The blue underlined bold text the auto- completed response from GPT-3, while the preceding text is the user-inputted prompt. In this 2-shot learning example, there initially incomplete example. On average, GPT-3 has low accuracy on high school mathematics questions.
(b) Performance on commonsense benchmark (Hel- laSwag), a linguistic understanding benchmark (Super- GLUE), and the massive multitask test. On previous benchmarks, smaller models start well above random chance levels and exhibit more continuous improve- ments with model size increases, but on our test, GPT-3 moves beyond random chance with the largest model.
instruction examples and one
specialized areas like law and ethics (Hendrycks al., 2020). The granularity and breadth of the subjects makes the benchmark ideal for identifying model's blind spots.
We find that meaningful progress on our benchmark has only become possible recent months. In particular, few-shot models up to 13 billion parameters (Brown et al., 2020) achieve random chance performance of 25% accuracy, but the 175 billion parameter GPT-3 model reaches a much higher 43.9% accuracy (see Figure lb). On the other hand, unlike human professionals GPT-3 does not excel at any single subject. Instead, we find that performance lopsided, with GPT-3 having almost 70% accuracy for its best subject but near-randon performance for several other subjects.
Our results indicate that while recent advances have been impressive, fatte-of-the-ant models still struggle at learning and applying knowledge from pretraining. The tasks with near-random accuracy include calculation-heavy subjects such as physics and mathematics and subjects related to human values such as law and morality. This second weakness is particularly concerning because will be important for future models to have a strong understanding of what is legal and what is ethical. Worryingly, we also find that GPT-3 does since its average confidence can be up to 24% off from its actual accuracy. We comprehensively evaluate the breadth and depth of a model's text understanding by covering numerous topics that humans are incentivized to learn. Since our test consists in 57 tasks, it can be used to analyze aggregate properties of models across tasks and track important shortcomings. The test and code available at github. heabubb.com/neddryks//sssst.
have an accurate sense of what does or does not know
2 RELATED WORK
Pretraining. The dominant paradigm in NLP is to pretrain large models on massive text corpora including educational books and websites. In the process, these models are exposed to information about a wide range of topics. Petroni et (2019) found that recent models learn enough information from pretraining that they can serve as knowledge bases. However, no prior work has comprehensively measured the knowledge models have across many real-world domains.
Until recently, researchers primarily used fine-tuned models on downstream tasks (Devlin et al., 2019). However, larger pretrained models like GPT-3 (Brown et al., 2020) have made possible achieve competitive performance without fine-tuning by using few-shot learning, which removes the need for a large line-tuning set. With the advent of strong zero-shot and few-shot learning, it is now possible to curate a diverse set of tasks for evaluation and remove the possibility of models on "spurious cues" (Geirhos et al., 2020; Hendrycks et al., 2019b) in a dataset to achieve high performance.
Benchmarks. Many recent benchmarks aim to assess a model's general world knowledge and basic reasoning ability by testing its "commonsensee"" A number of commonsense benchmarks have been
2

---

Published as a conference paper at ICLR 2021
As Seller, an encyclopedia salesman, approached the grounds on which Hermit's house was situated, he saw a sign that said, "No salesmen. Trespassers will be prosecuted. Proceed at your own risk." Although Seller had not been invited to enter, he ignored the sign and drove up the driveway toward the house. As he rounded curve, powerful explosive charge buried in the driveway exploded, and
Professional Law
Seller was injured. Can Seller recover damages from Hermit for his injuries?
(A) Yes, unless Hermit, when he planted the charge, intended only to deter, not harm, intruders.
(B) Yes, if Hermit was responsible for the explosive charge under the driveway.
(C) No, because Seller ignored the sign, which warned him against procoeding further.
X
(D) No, if Hermit reasonably feared that intruders would come and harm him or his family.
X
This task requires understanding detailed and dissonant scenarios, applying appropriate legal precedents, and choosing the correct explanation. The green checkmark is the ground truth.
Figure 2:
proposed in the past year, but recent models are already nearing human-level performance on several of these, including HellaSwag (Zellers et al., 2019), Physical IQA (Bisk et al., 2019), and CosmosQA (Huang et al., 2019). By design, these datasets assess abilities that almost every child has. In contrast, we include harder specialized subjects that people must study learn.
Some researchers have suggested that the future of NI.P evaluation should focus on Natural Language Generation (NIG) (Zellers et al., 2020), idea that reaches back to the Turing Test (Turing, 1950). However, NLG is notoriously difficult to evaluate and lacks a standard metric (Sai et al., 2020). Consequently, we instead create a simple-to-evaluate test that measures classification accuracy on multiple choice questions.
While several question answering benchmarks exist, they are comparatively limited in scope. Most either cover easy topics like grade school subjects for which models can already achieve strong performance (Clark al., 2018: Khot et al., 2019 Mihaylov et al., 2018; Clark et al., 2019), or are focused on linguistic understanding in the form of reading comprehension (Lai et al., 2017: Richardson et al., 2013). In contrast, we include wide range of difficult subjects that go far beyond linguistic understanding.
3 A MULTITASK TEST
We create massive multitask test consisting of multiple-choice questions from various branches of knowledge. The test spans subjects in the humanities, social sciences, hard sciences, and other areas that are important for some people to learn. There are 57 tasks in total, which is also the number of Atari games (Bellemare et al., 2013), all of which are listed in Appendix B. The questions in the dataset were manually collected by graduate and undergraduate students from freely available sources online. These include practice questions for tests such the Graduate Record Examination also includes questions designed for
and the United States Medical Licensing Examination. undergraduate courses and questions designed for readers of Oxford University Press books. Some tasks cover a subject, like psychology, but at a specific level of difficulty, such as "Elementary." "High School," "College," "Professional." " For example, the "Professional Psychology" task draws on questions from freely available practice questions for the Examination for Professional Practice in Psychology, while the "High School Psychology" task has questions like those from Advanced Placement Psychology examinations.
We collected 15908 questions in total, which we split into a few-shot development set, a validation set, and a test set. The few-shot development set has 5 questions per subject, the validation set may be used for selecting hyperparameters and is made of 1540 questions, and the test set has 14079 questions. Each subject contains 100 test examples at the minimum, which is longer than most exams designed to assess people.
Human-level accuracy on this test varies. Unspecialized humans from Amazon Mechanical Turk obtain 34.5% accuracy on this test. Meanwhile, expert-level performance can be far higher. For example, real-world test-taker human accuracy the 95th percentile is around 87% for US Medical Licensing Examinations, and these questions make up our "Professional Medicine" task. If we take the 95th percentile human test-taker accuracy for exams that build up our test, and if we make an educated guess when such information is unavailable, we then estimate that expert-level accuracy is approximately 89.8%.
Since our test aggregates different subjects and several levels of difficulty, we measure more than straightforward commonsense narrow linguistic understanding. Instead, measure arbitrary
3

---

Published as a conference paper at ICLR 2021
Microeconomics
One of the reasons that the government discourages and regulates monopolies is that (A) producer surplus is lost and consumer surplus is gained.
(B) monopoly prices ensure productive efficiency but cost society allocative efficiency.
monopoly firms do not engage in significant research and development.
consumer surplus is lost with higher prices and lower levels of output.
Figure 3: Examples from the Microeconomics task.
When you drop ball from rest accelerates downward at 9.8 m/s". If you instead throw it downward assuming air resistance its acceleration immediately after leaving your hand is (A) 9.8 m/s²
Conceptual Physies
more than 9.8 m/s"
less than 9.8 m/s²
(D) Cannot say unless the speed of throw is given.
In the complex z-plane, the set of points satisfying the equation z² =|/
Mathematic
College
(A) pair of points
(B) circle
half-line
(D) line
Figure 4: Examples from the Conceptual Physics and College Mathematics STEM tasks.
real-world text understanding. Since models are pretrained on the Internet, this enables us to test how well they can extract useful knowledge from massive corpora. Future models that use this test could be single models mixture of experts model. To succeed at test, future models should be well-rounded, possess extensive world knowledge, and develop expert-level problem solving ability. These properties make the test likely be an enduring and informative goalpost.
3.1 HUMANITIES
The humanities is a group of disciplines that make use of qualitative analysis and analytic methods rather than scientific empirical methods. Branches of the humanities include law, philosophy, history, and so on (Appendix B). Mastering these subjects requires a variety of skills. For example, legal understanding requires knowledge of how to apply rules and standards complex scenarios, and also provide answers with stipulations and explanations. We illustrate this in Figure 2. Legal understanding is also necessary for understanding and following rules and regulations, necessary capability to constrain open-world machine learning models. For philosophy, our questions cover concepts like logical fallacies, formal logic, and famous philosophical arguments. It also covers moral scenarios, including questions from the ETHICS dataset (Hendrycks et al., 2020) that test model's understanding of normative statements through predicting widespread moral intuitions about diverse everyday scenarios, Finally, our history questions cover a wide range of time periods and geographical locations, including prehistory and other advanced subjects.
3.2
SOCIAL SCIENCE
Social science includes branches knowledge that examine human behavior and society. Subject areas include economics, sociology. politics, geography, psychology, and See Figure 3 for an example question. Our economics questions include mieroeconomics, macroeconomics, and econometrics, and cover different types of problems, including questions that require mixture of world knowledge, qualitative reasoning, or quantitative reasoning. We also include important but more esoteric topics such as security studies in order test the boundaries of what is experienced and learned during pretraining, Social science also includes psychology, a field that may be especially important for attaining a nuanced understanding of humans.
3.3 SCIENCE, TECHNOLOGY, ENGINEERING, AND MATHEMATICS (STEM)
STEM subjects include physics, computer science, mathematics, and more. Two examples are shown in Figure 4. Conceptual physics tests understanding of simple physics principles and may be thought
4

---

Published as conference paper at ICLR 2021
A 33--car-oold man undergoes a radical thyroidectomy for thyroid cancer. During the operation, moderate hemorrhaging requires ligation of several vessels in the left side of the neck. Postoperatively, serum studies show a calcium concentration of mg/dL, albumin concentration of 4 g'dL, and parathyroid hormone concentration 200 pg/mL. Damage to which the following
Professional Medicine
vessels caused the findings in this patient?
(A) Branch of the costocervical trunk
(B) Branch of the external carotid artery
Branch of the thyrocervical trunk
Tributary of the internal jugular vein
Figure 5: A question from the Professional Medicine task.
of as a harder version of the physical commonsense benchmark Physical IQA (Bisk et al., 2019). We also test mathematical problem solving ability at various levels of difficulty, from the elementary to the college level. College mathematics questions, like those found on the GRE mathematics subject test, often require chains of reasoning and abstract knowledge. To encode mathematics expressions. we use La'TeX or symbols such and for multiplication and exponentiation respectively. STEM subjects require knowledge of empirical methods, fluid intelligence, and procedural knowledge.
3.4 OTHER
There a long tail of subjects that either do not neatly fit into any of the three preceding categories or for which there are not thousands of freely available questions. We put these subjects into Other. This section includes the Professional Medicine task, which has difficult questions that require humans many years of study to master. An example is depicted in Figure 5. This section also contains business topics like finance, accounting, and marketing, as well as knowledge of global facts. The latter includes statistics about poverty different countries over time, which may be necessary for having an accurate model the world internationally.
EXPERIMENTS
4.1 SETUP
Assessment and Models. To measure performance on our multitask test, compute the clas- sification accuracy across all examples and tasks. evaluate GPT-3 (Brown al., 2020) and UnifiedQA (Khashabi et al., 2020). For GPT-3 use the OpenAI API, which provides access to four model variants, "Ada," "Babbage," "Curie." " and "Davinci," which we refer to as "Small" (2.7 billion parameters), "Medium" (6.7 billion). "Largee" (13 billion) and "X-Large" (175 billion). UnifiedQA uses the T5 (Raffel et al., 2019) text-to-text backbone and fine-tuned on previously proposed question answering datasets (Lai et al., 2017), where the prediction the class with the highest token overlap with UnifiedQA's text output. Since UnifiedQA is fine-tuned on other datasets, we evaluate without any further tuning to assess its transfer accuracy. We also fine-tune RoBERTa-base. ALBERT-xxlarge, and GPT-2 on UnifiedQA training data and our dev+ set. We primarily focus on UnifiedQA and GPT-3 in the rest of this document, but additional discussion of RoBERTa, ALBERT, and GPT-221 is in Appendix A.
Model
Humanities
Social Science
STEM
Other
Average
Random Bascline
25,0
25.0
25.0
25.0
25.0
RoBERTa
27.9
28.8
27.0
27.7
27.9
ALBERT
27.2
25.7
27.7
27.9
27.1
GPT-2
32.8
33.3
30.2
33.1
32.4
UnifiedQA
45.6
56.6
40.2
54.6
48,9
GPT-3 Small (few-shot)
24.4
30.9
26.0
24.1
25.9
GPT-3 Medium (few-shot)
26.1
21.6
25.6
25.5
24.9
GPT-3 Large (few-shot)
27.1
25.6
24.3
26.5
26.0
GPT-3 X-Large (few-shot)
40.8
50.4
36.7
48.8
43.9
Table 1: Average weighted accuracy for each model on all four broad disciplines. All values percentages. Some models proposed in the past few months can move several percent points beyond random chance. GPT-3 uses few-shot learning and UnifiedQA is tested under distribution shift.
5

---

Published as a conference paper at ICLR 2021
Few-Shot Prompt. We feed GPT-3 prompts like that shown in Figure la. We begin each prompt with "The following are multiple choice questions (with answers) about [subject]." For zero-shot evaluation, we append the question to the prompt. For few-shot evaluation, we add up to 5 demonstration examples with answers to the prompt before appending the question. All prompts end with "Answer: The model then produces probabilities for the tokens "A," "B," "C," and "D," and we treat the highest probability option as the prediction. For consistent evalua-
Abstract Algebra
GPT-3
Anatomy
UnifiedOA
Astronemy
Random
Business Ethics
Clinical
College Biology
College Chemistry
Callege Comp Sel
College Mathematics
College Medicine
College Physics
Computer Security
Conceptual Physics
Econometrics
Electrical Engineering
Elementary Mathematics
tion, we create dev set with 5 fixed few-shot examples for each subject.
Logic
Glotal Facts
High School Diology
High School Chemistry
4.2 RESULTS
High School Comp Scl
High School European History
High
Geography
Model Size and Accuracy. We compare the few-shot accuracy of each GPT-3 size in Table 1. We find that the three smaller GPT-3 models have near random accuracy (around 25%). In contrast, we find that the X-Large 175 billion parameter GPT-3 model performs substantially better than random, with an accuracy of 43.9%. We also find qualitatively similar results in the zero-shot setting. While the smaller models have around 25% zero-shot accuracy, Figure 10 in Appendix shows that the largest GPT-3 model has a much higher zero-shot accuracy of about 37.7%. Brown et al. (2020) also observe that larger GPT-3 models perform better, though progress tends to be steadier. In Figure 1b we show that non-random accuracy on the multitask test emerged with recent large few-shot models compared to datasets that assess commonsense and linguistic understanding.
High School Gov't and Politics
High School Macroesonomies
High School Mathematics
High School Microeconomics
High School Physics
School Psychology
High School Statisties
High School US History
High Human Aging
History
Human Sexubiity
International
Jurisprudence
Logical Fallacies
Machine Leaming
Management
Marketing
Medical
Miscellannoos
Moral Disputes
Scenarios
Nutrition
Philesophy
Prehistory
Professional Accounting
Protessional Law
To test the usefulness of fine-tuning instead of few-shot learning, we also evaluate UnifiedQA models. UnifiedQA has the advantage of being fine-tuned on other question answering datasets, unlike GPT-3. We assess UnifiedQA by evalu- ating its transfer performance without any ad- ditional fine-tuning. The largest UnifiedQA model we test has 11 billion parameters, which is slightly smaller than GPT-3 Large. Neverthe- less, we show in Table that attains 48.9% Figure 6: GPT-3 (few-shot) and UnifiedQA results.
Professional Medicine
Professionel Psychology
Public Relations
Security Studies
Sociology
US Fomign Policy
Virology
World Religions
Accuracy (%)
accuracy. This performs better than the few-shot GPT-3 X-Large model, despite UnifiedQA have an order of magnitude fewer parameters. We also find that even the smallest UnifiedQA variant, with just 60 million parameters, has approximately 29.3% accuracy. These results suggest that while model size is a key component for achieving strong performance, fine-tuning also helps.
Comparing Disciplines. Using our test, discover that GPT-3 and UnifiedQA have lopsided performance and several substantial knowledge gaps. Figure 6 shows the accuracy of GPT-3 (few- shot) and UnifiedQA for all 57 tasks. It shows the both models for all tasks, with GPT-3's 's accuracy ranging from 69% for US Foreign Policy to 26% for College Chemistry. UnifiedQA does best marketing, with an accuracy of 82.5%.
below expert-level performance
Overall, models do poorly on highly procedural problems. Figure 6 shows that calculation- -eavy STEM subjects tend to have low accuracy compared to verbal subjects. For GPT-3, 9 out of the 10

---

Published as a conference paper at ICLR 2021
GPT-3 Zero-Shot Calibration
Declarative vs. Procedural Knowledge
60
Prompl and Completion:
Marketing
The order of operations or PEMDAS is
50
Parentheses Exponents Multiplication
Accuracy (%)
Division Addition Subtraction
40
Prompt and Completion:
(1 1) x 2
30
Figure 7: GPT-3's completion for two prompts testing knowledge of the order of operations. The blue underlined bold text is the autocompleted response from GPT-3. While it knows about the order of operations, it sometimes does not know how to apply its knowledge.
Formal Logic
20
20
30
40
50
60
Confidence (%)
Figure 8: GPT-3's confidence is poor estimator of its accuracy and can be off by up to 24%.
lowest-accuracy tasks are STEM subjects that emphasize mathematics calculations. We speculate that is in part because GPT-3 acquires declarative knowledge more readily than procedural knowledge. For example, many questions in Elementary Mathematics require applying the order of operations for arithmetic, which is described by the acronym PEMDAS (Parentheses Exponents Multiplication Division Addition Subtraction). In Figure 7, we confirm that GPT-3 is aware of the acronym PEMDAS. However, it does not consistently apply PEMDAS to actual problems. On the other hand, procedural understanding is not its only weak point. We find that some verbal tasks such as Moral Scenarios from Hendrycks et al. (2020) and Professional Law also have especially low accuracy.
Our test also shows that GPT-3 acquires knowledge quite unlike humans. For example, GPT-3 learns about topics in a pedagogically unusual order. GPT-3 does better College Medicine (47.4 (47.4%) and College Mathematics (35.0%) than calculation-heavy Elementary Mathematics (29.9%). GPT-3 demonstrates unusual breadth, but it does not master single subject. Meanhwhile suspect humans have mastery in several subjects but not as much breadth. In this way, our test shows that GPT-3 has many knowledge blindspots and has capabilities that are lopsided.
Calibration. We should not trust a model's prediction unless the model is calibrated, meaning that its confidence large neural networks are often miscalibrated (Guo et al., 2017), especially under distribution shift (Ovadia al., 2019). We evaluate the calibration of GPT-3 by testing how well its average confidence estimates its actual accuracy for each subject. We show the results in Figure 8, which demonstrates that GPT-3 is uncalibrated. In fact, its confidence is only weakly related to its actual accuracy in the zero-shot setting, with the difference between its accuracy and confidence reaching up 24% for some subjects. Another calibration measure is the Root Mean Squared (RMS) calibration error (Hendrycks et al., 2019a: Kumar et al., 2019). Many tasks have miscalibrated predictions, such as Elementary Mathematics which has a zero-shot RMS calibration error of 19.4%. Models are only somewhat more calibrated in the few-shot setting, shown in Appendix A. These results suggest that model calibration has wide room for improvement.
good estimate of the actual probability the prediction is correct. However,
5 DISCUSSION
Multimodal Understanding. While text is capable of conveying an enormous number of concepts about the world, many important concepts are conveyed mainly through other modalities, such as images, audio, and physical interaction (Bisk et al., 2020). Existing large-scale NLP models, such as GPT-3, do incorporate multimodal information, so we design our benchmark to capture a diverse array of tasks in a text-only format. However, models gain the ability to process multimodal inputs, benchmarks should be designed to reflect this change. One such benchmark could be a "Turk Test," " consisting of Amazon Mechanical Turk Human Intelligence Tasks. These are well-defined tasks that require models to interact with flexible formats and demonstrate multimodal understanding.
The Internet Training Set. A major distinction between our benchmark and previous multitask NLP benchmarks is that we do not require large training sets. Instead, we assume that models have acquired the requisite knowledge from reading quantities of diverse text from the Internet. This
7

---

Published as a conference paper at ICLR 2021
process is typically called pretraining, but it can be thought of as training in its own right, where the downstream evaluation is demonstrating whatever knowledge we would expect a human to pick up from reading the same text.
This motivates us to propose a methodological change so that models are trained more like how humans learn. While most previous machine learning benchmarks have models learn from large question bank, humans primarily learn new subjects by reading books and listening to others talk about the topic. For specialized subjects such as Professional Law, massive legal corpora are available, such as the 164-volume legal encyclopedia Corpus Juris Secundum, but there are fewer than 5,000 multistate bar exam questions available. Learning the entire law exclusively through small number of practice tests is implausible, so future models must learn more during pretraining.
For this reason we assess pretrained models in zero-shot, few-shot, or transfer setting and we provide a dev, val, and test set for each task. The dev set is used for few-shot prompts, the val set could be used for hyperparameter tuning, and the test set is used to compute the final accuracy. Importantly, the format of our evaluation is not identical to the format in which information is acquired during pretraining. This has the benefit of obviating concerns about spurious training set annotation artifacts (Geirhos et al., 2020; Hendrycks et al., 2019b) and is in stark contrast to the previous paradigm of identically distributed training and test sets. This change also enables collecting a much more extensive and diverse set of tasks for evaluation. We anticipate widespread as models improve at extracting information from diverse online sources.
methodology becoming more
Model Limitations. We find that current large-scale Transformers have wide room for improvement. They are notably poor at modeling human (disvaap,, as evident by the low performance on the Professional Law and Moral Scenarios tasks, For future systems be aligned with human values, high performance on these tasks is crucial (Hendrycks et al., 2020), so future research should especially aim to increase accuracy on these tasks. Models also have difficulty performing calculations, so much so that they exhibit poor performance on Elementary Mathematics and many other STEM subjects with "plug and chug" problems. Additionally, they do not match expert-level performance (90%) on any subject, so for all subjects it is subhuman. On average, models are only now starting to move beyond mandom-chance accuracy levels.
Addressing these shortcomings may be challenging. To illustrate this, we attempted to create a better Professional Law model by pretraining on specialized data but achieved only limited success. We collected approximately 2,000 additional Professional Law training examples. After fine-tuning RoBERTa- model (Liu et al., 2019) using this custom training set, our model attained 32.8% test accuracy. To test the impact of additional specialized training data, also had RoBERTa continue pretraining on approximately 1.6 million legal case summaries using Harvard's Law Library case law corpus case. 1 law, but after fine-tuning it only attained 36.1% accuracy. This suggests that while additional pretraining on relevant high quality text can help, it may not be enough to substantially increase the performance of current models.
It is unclear whether simply scaling up existing language models will solve the test. Current understanding indicates that 10x increase in model size must be accompanied by an approximate 5x increase in data (Kaplan et al., 2020). Aside from the tremendous expense in creating multi-trillio parameter language models, data may also become bottleneck, as there far less written about esoteric branches of knowledge than about everyday situations.
CONCLUSION
We introduced a new test that measures how well text models can learn and apply knowledge encountered during pretraining. By covering 57 subjects at varying levels of difficulty, the test assesses language understanding in greater breadth and depth than previous benchmarks. We found that it has recently become possible for models to make meaningful progress the test, but that state-of-the-art models have lopsided performance and rarely excel at any individual task. We also showed that current models are uncalibrated and have difficulty with tasks that require calculations. Worryingly, models also perform especially poorly on socially relevant subjects including morality and law. Our expansive test can help researchers pinpoint important shortcomings of models, making it easier to gain a clearer picture of state-of of-the-art capabilities.
8

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGEMENTS
We would like to thank the following for their helpful comments: Oyvind Tafjord, Jan Leike, David Krueger, Alex Tamkin, Girish Sastry, and Henry Zhu. DH supported by the NSF GRFP Fellowship and Open Philanthropy Project Fellowship. This research also supported by the NSF Frontier Award 1804794.
REFERENCES
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents (extended abstract). J. Artif. Intell. Res., 47:253-279,
Y. Bisk, R. Zellers, R. L. Bras, J. Gao, and Y. Choi. Piqa: Reasoning about physical commonsense in natural language, 2019.
Y. Bisk, A. Holtzman, J. Thomason, J. Andreas, Y. Bengio, J. Chai, M. Lapata, A. Lazaridou, J. May,
A. Nisnevich, N. Pinto, and J. Turian. Experience grounds language, 2020.
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, Gray, B. Chess, Clark, C. Berner, S. McCandlish, A. Radford, Sutskever, and D. Amodei. Language models are few-shot learners, 2020.
P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv,
2018.
P. Clark, O. Etzioni, D. Khashabi, T. Khot, B. D. Mishra, K. Richardson, A. Sabharwal, C. Schoenick, O. Tafjord, N. Tandon, S. Bhakthavatsalam, D. Groeneveld, M. Guerquin, and M. Schmitz. From 'f' to 'a' on the n.y. regents science exams: An overview of the aristo project. ArXiv, abs/1909.01958, 2019.
Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. ArXiv, abs/1880.04805, 2019.
R. Geirhos, J. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge, and F, A. Wichmann. Shortcut learning in deep neural networks, 2020.
C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger. On calibration of modern neural networks. ICML, 2017.
D. Hendrycks, M. Mazeika, and T. Dietterich. Deep anomaly detection with outlier exposure. ICLR, 2019a.
D. Hendrycks, Zhao, S. Basart, J. Steinhardt, and D. Song. Natural adversarial examples. ArXiv, 175714 2019b.
D. Hendrycks, C. Burns, Basart, A. Critch, J. Li, D. Song, and J. Steinhardt. Aligning with shared human values, 2020.
L. Huang, R. L. Bras, C. Bhagavatula, and Y. Choi. Cosmos qa: Machine reading comprehension with contextual commonsense reasoning, 2019.
J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei. Scaling laws for neural language models, 2020.
D. Khashabi, T. Khot, A. Sabharwal, O. Tafjord, P. Clark, and H. Hajishirzi. Unifiedqa: Crossing format boundaries with single qa system, 2020.
Khot, Clark, M. Guerquin, Jansen, and A. Sabharwal. Qasc: A dataset for question answering via sentence composition, 2019.
A. Kumar, P. Liang, and T. Ma. Verified uncertainty calibration, 2019.

---

Published as a conference paper at ICLR 2021
G. Lai, Q. Xie, H. Liu, Y. Yang, and E. Hovy. Race: Large-scale reading comprehension dataset from examinations, 2017.
Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut. Albert: A lite bert for self-supervised learning of language representations, ArXiv, abs/1909.1 1942, 2020.
Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and
V. Stoyanov. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11699 2019.
Mihaylov, P. Clark, Khot, and A. Sabharwal. Can a suit of armor conduct electricity? new dataset for open book question answering. In EMNLP, 2018.
Y. Ovadia, E. Fertig, J. Ren, Z. Nado, D. Sculley, S. Nowozin, J. V. Dillon, B. Lakshminarayanan, and J. Snoek. Can you trust your model's uncertainty? Evaluating predictive uncertainty under dataset shift. NeurIPS, 2019.
Petroni, Rocktäschel, P. Lewis, A. Bakhtin, Wu, A. H. Miller, and S. Riedel. Language models as knowledge bases?, 2019.
A. Radford, J. Wu, R. Child, D. Luan, Amodei, and I. Sutskever. Language models are unsupervised multitask learners. 2019.
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2019.
M. Richardson, C. Burges, and Renshaw. MCTest: A challenge dataset for the open-domain machine comprehension of text. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 193-203, Seattle, Washington, USA, Oct. 2013. Association for Computational Linguistics.
A. B. Sai, A. K. Mohankumar, and M. M. Khapra. A survey of evaluation metrics used for nlg systems. 2020.
A. Turing. Computing machinery and intelligence. 1950.
A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding, 2018
A. Wang, Y. Pruksachatkun, N. Nangia, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. Superglue: A stickier benchmark for general- purposse language understanding systems, 2019.
R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. Hellaswag: Can machine really finish your sentence?, 2019.
R. Zellers, A. Holtzman, E. Clark, L. Qin, A. Farhadi, and Y. Choi. Evaluating machines by their real-world language use, 2020.
10

---

Published as a conference paper at ICLR 2021
A ADDITIONAL ANALYSIS
This appendix includes figures with sorted results (Figure 9), few-shot examples vs. accuracy (Figure 10), and few-shot calibration (Figure 11). It also includes sections on fine-tuning, error analysis, and format sensitivity.
GPT-3 Results
UnifiedQA Results
Foreign Policy
Marketing
High School Psychology
High School Gov't and Politics
Miscellanaous
Management
Marketing
High School Psychology
High School Gov't and Politics
Business Ethics
High School Geography
jurisprudence
Computer Security
High School Geography
Management
International Law
High School World History
Sociclogy
International Law
Computer Security
World Religions
US Foreign Policy
jurisprudence
Miscelaneeus
Human Sexuality
High School World History
High School European History
High School History
Sociology
High School European Hlistory
High School US History
Logical Fallacies
Prehistory
Moral Disputes
Security Studies
World Religions
Philosophy
Public Relations
Human Aging
Philosophy
Bectrical Engineering
High School Microeconomics
Astronomy
Human Aging
Public Relations
High School Biology
High School Diology
Clinical Knowledge
Logical Fallacies
Prehistory
Clinical Knowledge
High School Comp Sei
Anatomy
Nutrition
College Medicine
Human Sexuality
Nutrition
Security Studies
Business Ethics
Professional Psychology
College Comp Scl
Bectrical Engineering
Virolngy
High School Macroeconomics
College Biclogy
Astronomy
Professional Psychoiogy
Professional Medicine
Moral Disputes
Virclogy
High School Microeconamics
College Medicine
High School Macroeconomics
Concoptual Physics
Medical Genetics
Colloge Comp Sci
High School Comp Scl
Anatomy
Global Facts
High School Physics
Conceptual Physics
College Biclogy
Professional Medicine
Professional Accounting
College Mathematics
Medical Genetics
Professional Law
Professional Law
Professional Accounting
Global Fatts
High School Chemistry
High School Chemistry
Lconometrics
College Mathemstics
Machine Learning
Elementary Mathematics
High School Statittics
College Chemistry
Abstract Algebra
High School Statétics
Elementary Mathematics
Machine Learning
Formal Logic
College Physics
High School Mathematics
High School Mathematics
High School Physics
Lconometrics
College Physies
Abstract Algebra
Moral Scenarios
Formal Logic
College Chemistry
Random Chance
Moral Scenarios
Random Chance
60
40
70
90
100
Accuracy (%)
Accuracy (%)
On the left are GPT-3 few shot accuracies for all of the 57 tasks. On the right are UnifiedQA transfer accuracies for all of the 57 tasks. For both models, capabilities are lopsided.
Figure 9:
A.1 1 ANALYSIS WITH MORE FINE-TUNED MODELS
We primarily analyzed models with more than 10 billion parameters in the main body of the paper. For this section, we analyze smaller models including RoBERTa-base (125 million parameters) (Liu
11

---

Published as a conference paper at ICLR 2021
al., 2019), ALBERR-xxxange (223 million parameters) (Lan al., 2020), and GPT-2 (1,558 million parameters) (Radford et al., 2019). Models are fine-tuned to predict one of four classes using the UnifiedQA MCQ questions and using dev+val set. We test our multitask test set.
We observe that these smaller models can attain overall accuracy of 27. 27.9%, with 27.9% accuracy for the humanities, 28.8% for social sciences, 27.0% for STEM, and 27.7% for other. ALBERT-xxlarge attains an accuracy 27.1%. with 27 27.2% accuracy for the humanities, 25.7% for the social sciences, 27.7% for STEM, and 27 27.9% for other. GPT-2 attains an accuracy of 32.4%, with 32.8% accuracy for the humanities, 33.3% for the social sciences, 30.2% for STEM, and 33.1% 1% for other.
accuracy. RoBERTa base attains
Compare this to UnifiedQA's smallest variant, which has just 60 million parameters and approximately 29.3% accuracy. obtains higher accuracy than RoBERTa and ALBERT, even though it has fewer parameters. This suggests that its larger pretraining dataset enables higher accuracy. Likewise, UnifiedQA with 3 billion parameters attains 43.7%, while the similarly sized GPT-2 model with 1.5 billion parameters attains 4% accuracy. This again suggests that T5's larger pretraining dataset size (and therefore UnifiedQA's pretraining dataset size) can increase accuracy.
A.2 ERROR ANALYSIS
We qualitatively analyze when GPT-3 makes high confidence mistakes. We find that while many of these mistakes were clearly wrong, many were mistakes that a human might make. For example, one question got wrong was "How many chromosomes do all human somatic cells contain?" The correct answer is 46, while few-shot GPT-3 predicted 23 with confidence 97.5%, This answer would have been correct if the question asked about the number of pairs of chromosomes. Similarly, many of its other high confidence mistakes were also correct answers to slightly different questions.
A.3 FORMAT SENSITIVITY
While different question formatting choices often lead to similar GPT-3 accuracies, we find that UnifiedQA is more sensitive. UnifiedQA's input format is of the form
QUESTION1 VAn (A) CHOICE1 (B) CHOICE2 (C) CHOICE3 (D) CHOICE4

where questions and choices are normalized and made lowercase. If we remove the 
 from the input, accuracy declines by several percentage points.
12

---

Published as a conference paper at ICLR 2021
GPT-3 Few-Shot Calibration
GPT-3 Multitask Accuracy vs. Number of Examples in Context
60
Accuracy (%)
45
Accuracy (%)
40
40
20
30
40
60
70
Confidence (%)
30
0-Shot 1-Shot 2-Shot 3-Shot 4-Shot 5-Shot Number Examples Context
Figure 11: While models are more calibrated in few-shot setting than zero-shot setting, they are still miscalibrated, with gap between accuracy and confidence reaching up 14%. Here the correlation between confidence and accuracy is 0.81, compared to 0.63 the zero-shot setting.
Figure 10: As the number few-shot instruction examples increases, the accuracy monotonically increases. Notably, zero-shot performance is only somewhat lower than 5-shot accuracy.
B TEST DETAILS
B. TASK DESCRIPTIONS AND EXAMPLES
We provide analysis of question length and difficulty in Figure 12. We list all tasks and the topics they test in Table 2. We also provide example for each task starting with Figure 14.
GPT-3 Average Question Length and Accuracy by Subject
GPT-3 Question Length and Correctness
1.0
Confidence of True Label
Subject Accuracy
0.6
0.4
0.2
0
500
1000 1500
2000
2500
3000
0.0
Question Length (Characters)
250
500
750
1000
1250
Average Question Length (Characters)
Figure 12: Figures on the relation between question difficulty and question length. For questions longer than a tweet (280 characters), the correlation between question length and true label confidence is slightly positive. This shows that longer questions are not necessarily harder.
B.2 EXACT QUESTION AND ANSWER CONTAMINATION
Since language models train on vast text corpora, there is some chance that they have seen the exact question and answer during pretraining. If they memorized the exact question and answer, then they would attain higher accuracy than their true ability. Likewise, question's 's entropy would be especially low if it were memorized. Memorized questions and answers should have low entropy and
13

---

Published as a conference paper at ICLR 2021
high accuracy. However, in Figure 13, that aceuracy and question entropy not positively correlated, suggesting that the test's low-entropy questions do not correspond to memorized (and thereby correctly predicted) answers. This suggests that our exact questions However, during pretraining models encountered text related to our questions through processing Wikipedia. We also note that most of our questions came from PDFs or websites where questions and answers are on separate pages.
not memorized.
See Brown et al. (2020) for previous discussion of contamination showing that the phenomena hardly affects performance. To reduce the probability that future models encounter exact questions during test-time, will provide list of question sources.
GPT-3 Zero-Shot Prompt Compression and Accuracy
GPT-3 Few-Shot
Prompt Compression and Accuracy
60
70
50
Accuracy (96
Accuracy (%)
40
50
40
30
20
20
-2.8
-2.6
-2.4
-2.
-2.0
-1.8
-2.4
-2.2
-2.0
-1.8
-1.6
-1.4
-1.2
Log Probability Per Token
Log Probability Per Token
Figure 13: The average log probability of the question (without answer) is strongly positively correlated with accuracy, all else equal. Each point corresponds to a task. Higher log probability indicates higher compression, and especially high log probability would suggest memorization. In the zero-shot question prompt, the correlation between average log probability and accuracy is -0.43, and for the few-shot setting the correlation is -0.56.
14

---

Published as a conference paper at ICLR 2021
Task
Tested Concepts
Supercategory
Abstract Algebra
Groups, rings, fields, vector spaces,
STEM
Anatomy
Central nervous system, circulatory system...
STEM
Astronomy
Solar system, galaxies, asteroids, Corporate responsibility, stakeholders, regulation, Spot diagnosis, joints, abdominal examination,
STEM
Business Ethics
Other
Clinical Knowledge
Other
College Biology
Cellular structure, molecular biology, ecology,
STEM
College Chemistry
Analytical, organic, inorganic, physical, Algorithms, systems, graphs, recursion, Differential equations, real analysis, combinatorics,... Introductory biochemistry, sociology, reasoning. Eleetromagneetsmm thermodynamics, special relativity, ... Cryptography, malware, side channels, fuzzing, Newton's laws, rotational motion, gravity, sound, Volatility, long-run relationships, forecasting, Circuits, power systems, electrical drives, Word problems, multiplication, remainders, rounding Propositions, predicate logic, first-order logic,.. Extreme poverty, literacy rates, life expectancy, ... Natural selection, heredity, cell cycle, Krebs cycle, Chemical reactions, ions, acids and bases, Arrays, conditionals, iteration, inheritance,
STEM
College Computer Science
STEM
College Mathematics
STEM
College Medicine
Other
College Physics
STEM
Computer Security
STEM
Conceptual Physics
STEM
Econometrics
Social Sciences
Electrical Engineering
STEM
Elementary Mathematics
STEM
Formal Logic
Humanities
Global Facts
Other
High School Biology
STEM
High School Chemistry
STEM
High School Computer Science
STEM
High School European History
Renaissance, reformation, industrialization.
Humanities
High School Geography
Population migration, rural land-use, urban processes.
Social Sciences
High School Gov't and Politics
Branches of govermment, civil liberties, political ideologies, Economic indicators, national income, international trade, Pre-algebra, algebra, trigonometry, calculus,
Social Sciences
High School Macroeconomics
Social Sciences
High School Mathematics
STEM
High School Microeconomics
Supply and demand, imperfect competition, market failure, Kinematics, energy, torque, fluid pressure,
Social Sciences
High School Physics
STEM
High School Psychology
Behavior, personality, emotions. learning,
Social Sciences
High School Statistics
Random variables, sampling distributions, chi-square tests, Civil War, the Great Depression, The Great Society, Ottoman empire, economic imperialism, World War I, Senescence, dementia, longevity, personality changes, Pregnancy, sexual differentiation, sexual orientation. Human rights, sovereignty, law of the sea, use of force, Natural law, classical legal positivism, legal realism, No true Scotsman, base rate fallacy, composition fallacy, SVMs, VC dimension, deep learning architectures, Organizing, communication, organizational structure, Segmentation, pricing, market research, ... Genes and cancer, common chromosome disorders, Agriculture, Fermi estimation, pop culture, ... Freedom of speech, addiction, the death penalty, Detecting physical violence, stealing, externalities,.. Metabolism, water-soluble vitamins, diabetes, Skepticism, phronesis, skepticism, Singer's Drowning Child. Neanderthals. Mesoamerica, extinction, stone tools,
STEM
High School US History
Humanities
High School World History
Humanities
Human Aging
Other
Human Sexuality
Social Sciences
International Law
Humanities
Jurisprudence
Humanities
Logical Fallacies
Humanities
Machine Learning
STEM
Management
Other
Marketing
Other
Medical Genetics
Other
Miscellaneous
Other
Moral Disputes
Humanities
Moral Scenarios
Humanities
Nutrition
Other
Philosophy
Humanities
Prehistory
Humanities
Professional Accounting
Auditing, reporting. regulation, valuation,
Other
Professional Law
Torts, criminal law, contracts, property, evidence,
Humanities
Professional Medicine
Diagnosis, pharmacotherapy. disease prevention, Diagnosis, biology and behavior, lifespan development,
Other
Professional Psychology
Social Sciences
Public Relations
Media theory, crisis management, intelligence gathering. Environmental security, terrorism, weapons of mass destruction, Socialization, cities and community, inequality and wealth,
Social Sciences
Security Studies
Social Sciences
Sociology
Social Sciences
US Foreign Policy
Soft power, Cold War foreign policy, isolationism, ... Epidemiology, coronaviruses, retroviruses, herpesviruses, Judaism, Christianity, Islam, Buddhism, Jainism,
Social Sciences
Virology
Other
World Religions
Humanities
Table 2: Summary of all 57 tasks.
15

---

Published as a conference paper at ICLR 2021
Find all e 23 such that Z [a(x)/(²² + c) is field.
(A)0 (B) (C)2 (D)3
Figure 14: An Abstract Algebra example.
What is the embryological origin of the hyoid bone?
(A) The first pharyngeal arch
(B) The first and second pharyngeal arches
(C) The second pharyngeal arch
(D) The second and third pharyngeal arches
Figure 15: An Anatomy example.
Why isn't there a planet where the asteroid belt is located?
(A) A planet once formed here but it was broken apart by catastrophic collision.
(B) There was not enough material in this part of the solar nebula to form planet.
(C) There was too much rocky material form a terrestrial planet but enough gascous material
to form jovian planet.
(D) Resonance with Jupiter prevented material from collecting together form a planet.
Figure 16: An Astronomy example.
Three contrasting tactics that CSO's can engage to meet their aims are
which typi- which may involve physically attacking
cally involves research and communication,
company's 's operations or
often involving some form of (A) Non-violent direct action, Violent direct action, Indirect action, Boycott
(B) Indirect action, Instrumental action, Non-violent direct action, Information campaign
(C) Indirect action, Violent direct action, Non-violent direct-action Boycott.
(D) Non-violent direct action, Instrumental action, Indirect action, Information campaign
Figure 17: A Business Ethics example.
How many attempts should you make to cannulate a patient before passing the job on to a senior colleague?
(A) 4 (B)
(C) 2 (D)
Figure 18: A Clinical Knowledge example.
In given population, 1 out of every 400 people has a cancer caused by completely recessive allele, b. Assuming the population is in Hardy- Weinnbrr equilibrium, which of the following is the expected proportion of individuals who carry the b allele but are not expected to develop the cancer?
(A) 1/400
(B) 19/400
(C) 20/400
(D) 38/400
Figure 19: College Biology example.
Which of the following statements about the lanthanide elements is NOT true?
(A) The most common oxidation state for the lanthanide elements is +3.
(B) Lanthanide complexes often have high coordination numbers 6).
(C) All of the lanthanide elements react with aqueous acid to liberate hydrogen.
(D) The atomic radii of the lanthanide elements increase across the period from La to Lu.
Figure 20: A College Chemistry example.
16

---

Published as a conference paper at ICLR 2021
Consider a computer design in which multiple processors, each with a private cache memory, share global memory using single bus. This bus is the critical system resource. Each processor can execute one instruction every 500 nanoseconds as long as memory references are satisfied by its local cache. When cache miss occurs, the processor is delayed for additional 2,000 nanoseconds. During half of this additional delay, the bus is dedicated to serving the cache miss. During the other half, the processor cannot continue, but the bus free service requests from other processors. On average, each instruction requires 2 memory references. On average, cache misses occur on percent of references. What proportion of the capacity of the bus would single processor consume, ignoring delays due to competition from other processors?
(A) 1/50 (B) 1/27
(C) 1/25 (D) 2/27
Figure 21: A College Computer Science example.
Let A be a real x 2 matrix. Which the following statements must be true?
All of the entries of A are nonnegative.
II. The determinant of A is nonnegative.
III. If has two distinct eigenvalues, then A has distinct eigenvalues.
(A) only
(B) II only
(C) III only (D) II and III only
Figure 22: A College Mathematics example.
In genetic test of newborn, rare genetic disorder is found that has X-linked recessive transmission. Which of the following statements is likely true regarding the pedigree of this disorder?
(A) All descendants on the maternal side will have the disorder.
(B) Females will be approximately twice as affected as males this family,
(C) All daughters of an affected male will be affected.
(D) There will be equal distribution of males and females affected.
Figure 23: A College Medicine example.
One end of a Nichrome wire of length 2L and cross-sectional area A is attached to an end of another Nichrome wire of length L and cross- sectional area 2A. If the free end of the longer wire is at an electric potential of 8.0 volts, and the free end of the shorter wire is at an electric potential of 1.0 volt, the potential the junetion of the two wires is most nearly equal to
(A) 2.4 V
(B) 3.3
(C) 4.5 V
(D) 5.7 V
Figure 24: A College Physics example.
Why it that anti-virus scanners would not have found an exploitation of Heartbleed?
(A) It's vacuous question: Heartbleed only reads outside buffer, so there is no possible exploit
(B) Anti-virus scanners tend to look for viruses and other malicious
(C) Heartbleed attacks the anti-virus scanner itself
(D) Anti-virus scanners tend to look for viruses and other malicious code, but Heartbleed
exploits steal secrets without injecting any code
Figure 25: A Computer Security example.
A model airplane flies slower when flying into the wind and faster with wind at its back. When launched right angles the wind, cross wind, its groundspeed compared with flying in still
air is
(A) the same
(B) greater
(C) less
(D) either greater less depending on wind speed
Figure 26: A Conceptual Physics example.
17

---

Published as a conference paper at ICLR 2021
Consider the following AR(1) model with the disturbances having zero mean and unit variance y/ 0.2 + 0.43/-1 Me The (unconditional) mean of y will be given by (A) 0.2
(B) 0.4 0.5 (D) 0.33
Figure 27: An Econometrics example.
A point pole has a strength of 10-4 weber. The force in newtons on a point pole of 4x x 1.5 x 10-4 weber placed distance of 10 from it will be
(A) N. (B) N. (C) N. (D) 3.75
Figure 28: An Electrical Engincering example.
A total 30 players will play basketball at park. There will be exactly 5 players on each team. Which statement correctly explains how to find the number of teams needed?
(A) Add 5 to 30 to find 35 teams.
Divide 30 by
find teams.
(C) Multiply 30 and 5 to find 150 teams.
(D) Subtract 5 from 30 to find 25 teams.
Figure 29: An Elementary Mathematics example.
Determine whether the statements are logically equivalent or contradictory. If neither, determine whether they are consistent or inconsistent.
E D (F. E) and ~ E
(A) Logically equivalent
(B) Contradictory
(C) Neither logically equivalent nor contradictory, but consistent
(D) Inconsistent
Figure 30: A Formal Logic example.
As 2017, how many of the world's 1-year-old children today have been vaccinated against some disease?
(A) 80%
(B) 60%
(C) 40%
(D) 20%
Figure 31: A Global Facts example.
Homologous structures are often cited evidence for the process of natural selection. All of the following are examples of homologous structures EXCEPT
(A) the wings of bird and the wings of bat
(B) the flippers of a whale and the arms of man
(C) the pectoral fins of a porpoise and the flippers seal
(D) the forelegs of insect and the forelimbs of dog
Figure 32: A High School Biology example.
From the solubility rules, which of the following is true?
(A) All chlorides, bromides, and iodides are soluble
(B) All sulfates are soluble
(C) All hydroxides are soluble
(D) All ummoniim--cntaiinng compounds are soluble
Figure 33: A High School Chemistry example.
18

---

Published as a conference paper at ICLR 2021
A list of numbers has n elements, indexed from to n. The following algorithm intended to display the number of clements in the list that have value greater than 100. The algorithm uses the variables count and position. Steps 3 and 4 are missing.
Step I: Set count to 0 and position to 1.
Step 2: If the value of the element at index position is greater
than 100, increase the value of count by I.
Step 3: (missing step)
Step (missing step)
Step 5: Display the value of count.
Which of the following could be used to replace steps 3 and 4 that the algorithm works as intended?
(A) Step Increase the value of position by 1.
Step 4: Repeat steps 2 and 3 until the value of count is greater than 100.
(B) Step 3: Increase the value of position by
Step 4: Repeat steps and 3 until he value of position is greater than
(C) Step 3: Repeat step 2 until the value of count is greater than 100.
Step 4: Increase the value of position by 1.
(D) Step 3: Repeat step 2 until the value of position is greater than n.
Step 4: Increase the value of count by 1.
Figure 34: A High School Computer Science example.
This question refers to the following information.
Albeit the king's s Majesty justly and rightfully is and ought to be the supreme head of the Church of England, and so is recognized by the clergy of this realm in their convocations, yet nevertheless, for corroboration and confirmation thereof, and for increase of virtue in Christ's religion within this realm of England, and to repress and extirpate all errors, heresies, and other enormities and abuses heretofore used in the same, be it enacted, by authority of this present Parliament, that the king, our sovereign lord, his heirs and successors, kings of this realm, shall be taken, accepted, and reputed the only supreme head in earth of the Church of England, called Anglicans Ecclesia; and shall have and enjoy, annexed and united to the imperial crown of this realm, as well the title and style thereof, all honors, dignities. preeminences, jurisdictions, privileges, authorities, immunities, profits, and commodities to the said dignity of the supreme head of the same Church belonging and appertaining: and that our said sovereign lord, his heirs and successors, kings of this realm, shall have full power and authority from time to time to visit, repress, redress, record, order, correct, restrain, and amend all such errors, heresies, abuses, offenses, contempts, and enormities, whatsoever they be, which by any manner of spiritual authority or jurisdiction ought or may lawfully be reformed, repressed, ordered, redressed, corrected, restrained, or amended, most to the pleasure of Almighty God, the increase of virtue in Christ's 's religion, and for the conservation the peace, unity, and tranquility of this realm; any usage, foreign land, foreign authority, prescription, or any other thing or things to the contrary hereof notwithstanding. English Parliament, Act of Supremacy, 1534
From the passage, one may infer that the English Parliament wished argue that the Act of Supremacy would
(A) give the English king new position of authority
(B) give the position of head of the Church of England to Henry VIII alone and exclude his heirs
(C) establish Calvinism as the one true theology in England
(D) end various forms of corruption plaguing the Church in England
Figure 35: A High School European History example.
During the third stage of the demographic transition model, which of the following is true?
(A) Birth rates increase and population growth rate is less rapid.
(B) Birth rates decline and population growth rate is less rapid.
(C) Birth rates increase and population growth rate increases.
(D) Birth rates decrease and population growth rate increases.
Figure 36: A High School Geography example.
19

---

Published as a conference paper at ICLR 2021
Which of the following best states argument made by James Madison in The Federalist number
10?
(A) Honest politicians can prevent factions from developing.
(B) Factions are more likely to occur in large republics than in small ones.
(C) The negative effects of factionalism can be reduced by a republican government.
(D) Free elections are the people's 's best defense against factionalism.
Figure 37: High School Government and Politics example.
Which of the following not included in the U.S. GDP?
(A) The U.S. military opens a new base in foreign country with 1000 U.S. personnel.
(B) Japanese consumers buy thousands of CDs produced the United States.
(C) An American pop singer performs sold-out concert in Paris.
(D) A French theatrical production tours dozens American cities.
Figure 38: A High School Macroeconomics example.
Joe was in charge of lights for dance. The red light blinks every two seconds, the yellow light every three seconds, and the blue light every five seconds. If we include the very beginning and very end of the dance, how many times during seven minute dance will all the lights come at the same time? (Assume that three lights blink simultaneously at the very beginning of the dance.)
(A)
(B) 15
(C)
(D)
Figure 39: A High School Mathematics example.
If the government subsidizes producers in a perfectly competitive market, then
(A) the demand for the product will increase
(B) the demand for the product will decrease
(C) the consumer surplus will increase
(D) the consumer surplus will decrease
Figure 40: High School Microcconomics example.
A point charge, == +1 mC, fixed at the origin. How much work required to move charge,
Q= +8 μC, from the point (0, 4 meters) to the point (3 meters, 0)?
(A) 3.5
(B) 6.0 J
(C) 22.5
(D) J0
Figure 41: High School Physics example.
While swimming in the ocean, Ivan is frightened by a dark shadow in the water even before he has the chance to identify what the shadow is. The synaptic connections taking place during this incident of fright are best described by which of the following?
(A) Messages are sent from the thalamus directly to the amygdala.
(B) Messages are sent from the thalamus to the "what" and "where" pathways.
(C) Messages are sent from the parasympathetic nervous system to the cerebral cortex.
(D) Messages are sent from the frontal lobes to the pituitary gland.
Figure 42: A High School Psychology example.
20

---

Published as a conference paper at ICLR 2021
Jonathan obtained score of 80 statistics exam, placing him at the 90th percentile. Suppose five points are added to everyone's 's score, Jonathan's new score will be at the
(A) 80th percentile.
(B) 85th percentile.
(C) 90th percentile.
(D) 95th percentile.
Figure 43: A High School Statistics example.
This question refers to the following information.
"'occity in every state is blessing, but government in its best state is but a necessary evil; in its worst state an intolerable one; for when we suffer, or are exposed to the same miseries by government, which we might expect in country without government, our calamity heightened by reflecting that we furnish the means by which we suffer. Government, like dress, is the badge of lost innocence; the palaces of kings are built on the ruins of the bowers of paradise. For were the impulses of conscience clear, uniform, and irresistibly obeyed, man would need no other lawgiver; but that not being the case, finds necessary to surrender up a part of his property to furnish means for the protection of the rest; and this he is induced to do by the same prudence which in every other case advises him out two evils to choose the least. Wherefore, security being the true design and end of government, it unanswerably follows that whatever form thereof appears most likely to ensure it to us, with the least expense and greatest benefit, is preferable to all others."
Thomas Paine, Common Sense, 1776
Which of the following "miseries" alluded above were most condemned by Anti-Federalists of the Resoutevolutionrry era?
(A) Organized response to Bacon's Rebellion.
(B) Federal response to Shays's Rebellion.
(C) Federal response to the Whiskey Rehellion.
(D) Federal response to Pontiac's Rebellion.
Figure 44: A High School US History example.
This question refers to the following information.
"The real grievance of the worker is the insecurity of his existence; he is not sure that he will always have work, he is not sure that he will always be healthy, and he foresees that he will one day be old and unfit to work. If he falls into poverty, even if only through a prolonged illness, he is then completely helpless, left to his own devices, and society does not currently recognize any real obligation towards him beyond the usual help for the poor. even he has been working all the time ever so faithfully and diligently. The usual help for the poor, however, leaves a lot to be desired, especially large cities, where it very much worse than in the country." Otto von Bismarck, 1884
Otto von Bismarck likely made this speech in reaction to which of the following issues?
(A) Social acceptance of child labor.
(B) Declining life expectancy in Germany.
(C) Criticisms of German trade tariffs.
(D) Negative effects attributed to industrial capitalism.
Figure 45: A High School World History example.
All other things being equal, which of the following persons is more likely to show osteoporosis?
(A) An older Hispanic American woman
(B) An older African American woman
(C) An older Asian American woman
(D) An older Native American woman
Figure 46: A Human Aging example.
21

---

Published as a conference paper at ICLR 2021
Morning sickness is typically a problem:
(A) during the first trimester
(B) during the second trimester
(C) during the third trimester
(D) all through the pregnancy
Figure 47: A Human Sexuality example.
Would a reservation to the definition of torture in the ICCPR be acceptable in contemporary practice?
(A) This is an acceptable reservation if the reserving country's legislation employs different definition
(B) This is an unacceptable reservation because it contravenes the object and purpose of
the ICCPR
(C) This is unacceptable reservation because the definition of torture in the ICCPR is consistent with customary international law
(D) This is acceptable reservation because under general international law States have the right
to enter reservations to treaties
Figure 48: An International Law example.
Which position does Rawls claim is the least likely be adopted by the POP (people in the original position)?
(A) The POP would choose equality above liberty.
(B) The POP would opt for the 'maximin' strategy.
(C) The POP would opt for the 'difference principle.'
(D) The POP would reject the 'system of natural liberty.'
Figure 49: A Jurisprudence example.
John Stuart Mill: Each person's happiness is a good to that person, and the general happiness, therefore, good to the aggregate of all persons.
(A) Fallacy of Composition
(B) Fallacy of Division
(C) Gambler's Fallacy
(D) Equivocation
Figure 50: A Logical Fallacies example.
A 6-sided die is rolled 15 times and the results are: side 1 comes up 0 times; side 2: 1 time; side 3: 2 times; side 4: 3 times; side 5: 4 times; side 6: times, Based on these results, what is the probability of side 3 coming up when using Add-1 Smoothing?
(A) 2/15
(B) 1/7 (C) 3/16 (D) 1/5
Figure 51: A Machine Learning example.
According to Lewin, Lippet and White's 1939 experiment, which form of leadership produced the work from participants?
(A) Laissez-faire
(B) Democratic
(C) Authoritarian
(D) A mix of laissez-faire and democratic
Figure 52: A Management example.
22

---

Published as a conference paper at ICLR 2021
The single group within society that is most vulnerable reference group influence is:
(A) The older consumer who feels somewhat left out of things.
(B) The married women, many whom feel need for stability in their lives.
(C) New immigrants who really want to assimilate into their new culture.
(D) Children, who base most of their buying decisions on outside influences.
Figure 53: A Marketing example.
Which of the following conditions does not show multifactorial inheritance?
(A) Pyloric stenosis
(B) Schizophrenia
(C) Spina bifida (neural tube defects)
(D) Marfan syndrome
Figure 54: Medical Genetics example.
If each of the following meals provides the same number of calories, which meal requires the
most land to produce the food?
(A) Red beans and rice
(B) Steak and a baked potato
(C) Corn tortilla and refried beans
(D) Lentil soup and brown bread
Figure 55: A Miscellaneous example.
In response Sandel's "social justice" argument, Kamm argues that
(A) even if we were able to enhance ourselves or others, we would not thereby be obligated to do s0.
(B) there is a difference between (i) choosing that one have certain trait and (ii) being responsible
for bearing the costs of that choice.
(C) having the option to enhance will lead to fewer instances of people who require the assistance
of others.
(D) all of the above
Figure 56: A Moral Disputes example.
For which of these two scenarios does the main character (who uses l/me/my) do something clearly morally wrong?
Scenario I drive very fast when I'm on racetrack. Scenario 21 drive very fast when I'm in the suburbs.
(A) Wrong, Wrong
Wrong, Not wrong
(C) Not wrong, Wrong
(D) Not wrong, Not wrong
Figure 57: A Moral Scenarios example. The formatting of this task hinders UnifiedQA performance substantially.
Which of the following is the most plausible explanation for the protective effect dietary fibre against cancer of the colon?
(A) Propionic acid, formed during colonic fibre fermentation inhibits liver fatty acid synthesis
(B) Butyric acid, formed during colonic fibre fermentation stimulates "silencing" of the SLC5A8
tumour suppressor gene
(C) Butyric acid, formed during colonic fibre fermentation stimulates anti-oxidant defences
in the colon
(D) None of these options are correct
Figure 58: A Nutrition example.
23

---

Published as a conference paper at ICLR 2021
According to Moore's "ideal utilitarianism," the right action is the one that brings about the greatest amount of:
(A) pleasure.
happiness.
(C) good.
(D) virtue.
Figure 59: A Philosophy example.
Researchers now believe that the decline of the Maya was caused chiefly by:
(A) cataclysm of some kind, such as an earthquake, volcano, or tsunami.
(B) ecological degradation resulting from slash-and-burn farming techniques.
(C) endless wars between neighboring Mayan city-states.
(D) practices of interbreeding that led to a steep rise in congenital disorders.
Figure 60: A Prehistory example.
Krete is an unmarried taxpayer with income exclusively from wages. By December 31, year Krete's employer has withheld $16,000 in federal income taxes and Krete has made estimated tax payments. On April 15, year 2, Krete timely filed for an extension request to file her individual tax return, and paid $300 additional taxes. Krete's year tax liability was $16,500 when she timely filed her return on April 30, year 2, and paid the remaining tax liability balance. What amount would be subject to the penalty for underpayment of estimated taxes?
(A) $0
(B) $500
(C) $1,650
(D) $16,500
Figure 61: A Professional Accounting example.
The night before his bar examination, the examinee's next-door neighbor was having party. The music from the neighbor's home was so loud that the examinee couldn't fall asleep. The examince called the neighbor and asked her to please keep the noise down. The neighbor then abruptly hung up. Angered, the examinee went into his closet and got a gun. He went outside and fired bullet through the neighbor's 's living room window. Not intending to shoot anyone, the examinee fired his gun at such an angle that the bullet would hit the ceiling. He merely wanted to cause some damage the neighbor's 's home to relieve his angry rage. The bullet, however, ricocheted off the ceiling and struck a partygoer in the back, killing him. The jurisdiction makes it misdemeanor to discharge firearm public. The examinee will most likely be found guilty for which of the following crimes in connection to the death of the partygoer?
(A) Murder.
(B) Involuntary manslaughter.
(C) Voluntary manslaughter.
(D) Discharge of firearm in public.
Figure 62: A Professional Law example.
24

---

Published as a conference paper at ICLR 2021
A 63-year-old man is brought the emergency department because of 4-day history of in- creasingly severe left leg pain and swelling his left calf. He also has a 1-month history of increasingly severe upper midthoracic back pain. During this time, he has had 9-kg (20-1b) weight loss despite no change in appetite. He has no history of major medical illness. His only medication is ibuprofen. He is 180 em ft 11 in) tall and weighs 82 kg (180 lb); BMI is 25 kg/m2. His vital signs are within normal limits. On examination. lower extremity pulses are palpable bilaterally. The remainder of the physical examination shows no abnormalities. An x-ray of the thoracic spine shows no abnormalities. A CT scan of the abdomen shows 3-cm mass in the body of the pancreas; there are liver metastases and encasement of the superior mesenteric artery. Ultrasonography of the left lower extremity shows femoropopliteal venous clot. Which of the following is the most likely cause of this patient's 's symptoms?
(A) Carcinoid syndrome
(B) Hyperroaguulability from advanced malignancy
(C) Multiple endocrine neoplasia
(D) Splenic artery aneurysm and embolic disease of the left lower extremity
Figure 63: A Professional Medicine example.
The technique that is most likely to produce immediate improvement in the behavior of child who hits others and rips up schoolbooks is
(A) combination of reinforcement for appropriate behavior and mild punishment for in- appropriate behavior
(B) differential positive reinforcement
(C) loss of privileges for misbehaving
(D) shaping
Figure 64: A Professional Psychology example.
You work for a utility company that is building biomass plant in the community. Your employer asks you to give a series of community talks about the plant and future operations. You visit the plant several hours before you are due to give a speech that has been prepared by your immediate supervisor. During the tour of the plant, you discover several claims in the speech are not true. What do you do?
(A) Write and deliver a new speech that you know is entirely correct.
(B) Cancel all speeches until you and your supervisor get the information straight.
(C) Deliver the speech as prepared and discuss the inaccuracies with your supervisor afterward.
(D) Address the inaccuracies with your supervisor immediately and make the necessary
corrections before giving the speech.
Figure 65: A Public Relations example.
25

---

Published as a conference paper at ICLR 2021
The Chemical Weapons Convention (CWC) prohibited the possession or deployment of chemical weapons; however failed to implement stipulations that would require signatories to declare their existing stocks of chemical weapons, to identify facilities that were once involved in chemical production, or to announce when their existing stocks would be destroyed.
(A) The Chemical Weapons Convention (CWC) prohibited the possession deployment of chemical weapons; however it failed to implement stipulations that would require signatories to declare their existing stocks of chemical weapons, to identify facilities that were once involved in chemical production, or to announce when their existing stocks would be destroyed.
(B) The CWC made some important developments regarding the and possession of chemical weapons and the destruction of existing stockpiles. However, the treaty failed to establish an independent body empowered with the capacity check treaty compliance. Lack of supra-state authority has undermined the ability to enforce those developments. Given the anarchical nature of international society it may be in the national security interest retain stocks.
(C) Chemical weapons continue to exert a determining influence on international society. As early as the 1970s military strategists were convinced of the deterrence effects chemical weapons could have, comparable to the second strike survival logic of nuclear deterrence. The preferences of strategists resulted in continued manufacture and stockpiling of weapons creating an international crisis of stability.
(D) While the CWC has been ratified by the majority of international society, some nations with large chemical capability at their disposal have yet to enter into the treaty. However, to some analysts the destructive military potential would be limited, having a moderate effect on a well-equipped army in conventional warfare. Chemical arsenal essentially falls under the category of the "poor mans" weaponry, being simplistic and inexpensive whilst having limited military utility. However, the concern remains of the prospective impact terrorist chemical attack could have on civilian populations.
Figure 66: Security Studies example.
Which of the following statements most closely corresponds with differential association theory?
(A) If all of your friends jumped off a bridge, I suppose you would too.
(B) You should be proud to be a part of this organization.
(C) If the door is closed, try the window.
(D) Once a thief, always thief.
Figure 67: Sociology example.
Why did Congress oppose Wilson's proposal for the League of Nations?
(A) It feared the League would encourage Soviet influence in the US
(B) It feared the League would be anti-democratic
(C) It feared the League would commit the US to an international alliance
(D) Both and b
Figure 68: A US Foreign Policy example.
An observational study in diabetics assesses the role of an increased plasma fibrinogen level on the risk of cardiac events. 130 diabetic patients followed for years to assess the development of acute coronary syndrome. In the group of 60 patients with a normal baseline plasma fibrinogen level, 20 develop acute coronary syndrome and 40 do not. In the group of 70 patients with a high baseline plasma fibrinogen level, 40 develop acute coronary syndrome and 30 do not. Which of the following is the best estimate of relative risk in patients with high baseline plasma fibrinogen level compared to patients with normal bascline plasma fibrinogen level?
(A) (40/30)((0/40)
(B) (40*40)((0030)
(C) (40*70)((20600)
(D) (40/70)((00/60)
Figure 69; A Virology example.
26

---

Published as a conference paper at ICLR 2021
The Great Cloud Sutra prophesied the imminent arrival of which person?
(A) Maitreya (Milo)
(B) The Buddha
Zhou Dunyi
(D) Wang Yangming
Figure 70: World Religions example.
27


Reviews:
[{"review_id": "yVVwi_zBSse", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n \nThe paper proposes a benchmark for NLP models. The purpose of this test is to measure the model's knowledge in 57 topics covered by approx 15000 tasks in total, each formulated as a closed-form question in zero-shot and few-shot settings. Most of the tasks were taken from different human examination sets. Then, the paper provides results of experiments with the latest (GPT-3 and T5 based) models along with some quantitative and qualitative observations.\n\n\nPros:\n+ Such a dataset of questions may be useful in the future if it will be published.\n+ The authors show that in the case of GPT-3, as with earlier similar models, the model's confidence is not a good estimate of the actual probability of the prediction is correct.\n\nCons and questions:\n- The paper compares only models of two different architectures, while the proposed format for comparing the probabilities of 4 tokens allows on to test even models of the BERT family.\n- I believe it's important to understand the human level in such benchmarks. Moreover, it would be interesting to see several human baselines with different levels of education.\n- It is known that the ability to answer zero-shot/few-shot questions depends on the size of the model, but the inability to answer in the zero-shot format does not necessarily mean a lack of necessary knowledge in the model. Thus, the proposed approach is biased towards larger models that are just better able to work in zero-shot mode.\n- Despite the fact that the UnifiedQA model is superior in quality to GPT-3, most of the results are devoted to the GPT-3 model.\n- The uneven success of models for different topics can possible be explained by several objective reasons (the average length of a topic question in tokens, the average frequency of topic question tokens in the corpus, the share of topic documents in the training sample, and so on). At least some of them can be checked within the paper.\n- It's unclear if the authors are going to publish the questionnaire.\n\nOverall, I vote for rejecting. I like the idea of a comprehensive questionnaire as an NLP benchmark but besides the proposed set of questions the paper includes only the results for two different architectures and some debatable hypotheses on the reasons for such results.\n\nUPDATE:\nAuthors, thanks for your updates.\nSome of cons are gone, and the paper is better now, but my main concern stays: it's unclear if the results are about the problem solving ability or the zero-shot learning ability. Thus, I corrected my score to from 4 to 5.", "extra_scores": {}, "extra_text": {}}, {"review_id": "ttAZhdybjH2", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper describes a dataset consisting of ~14k multiple-choice questions drawn from many different fields across the humanities and science as well as professional disciplines such as law and medicine. It presents results for GPT-3 models (LMs trained on text corpora with document context) of different scales, as well as for the UnifiedQA model (seq2seq model trained on various QA datasets). Performance of these models is well below their performance on other benchmarks: not above chance for the smaller GPT-3 models, and under 50% average accuracy for the best models.\n\nThe paper has a lot of good features: it’s obviously great to have a broad, challenging, large-scale dataset that aims to separate human from model performance. The lack of fine-tuning is a plus, as is having an objective evaluation procedure via multiple-choice answers. The calibration results for GPT-3 are very interesting.\n\nHowever, I wonder about the value of a test that covers so many specialized areas of knowledge. A model that achieved expert performance (90% is suggested) across the board would clearly be superhuman, since no one is an expert in all these areas. Even a model that achieved expert performance in a few areas would probably be doing better than the average person. So instead of actually separating model performance from human performance, this benchmark might amount to just moving the goalposts about what constitutes human performance. To calibrate, it would have been helpful (albeit expensive, I realize) to see human scores from both experts and non-experts on this data.\n\nAnother problem is that the benchmark is likely to reward exposure to specialized domain data. Some tasks may benefit from this more than others, depending on the extent to which they are already represented in general-domain training corpora, and the extent to which they reward rote learning as opposed to true generalization. So it’s possible that the benchmark can be gamed by discovering these ‘easy’ tasks and doing aggressive data mining. The authors provide some evidence that this kind of approach isn’t likely to be fruitful for law, but that’s only one domain, and their experiment isn’t strictly comparable with, for instance, retraining GPT-3 on a corpus that includes many more legal documents. Ideally the paper would have included more evidence about the relative difficulty of tasks, linked to the amount of relevant training data for the models tested. Another option would have been to include only ‘hard’ tasks. But that begs the question about the value of using specialized tasks in the first place, as opposed to just training on general text and (more)  assiduously testing for world knowledge that’s unlikely to be explicitly represented (to borrow an example from the T5 paper: can a tuba fit into a backpack?).\n\nDetailed comments and questions:\n- Figure 2 runs up against the previous text.\n- It would be good to merge UnifiedQA results (from figure 9) into figure 6.\n- Is the plan to release this dataset publicly? There is no mention of this in the paper.\n- To ensure a fair comparison, it would be interesting to run GPT-3 in the same text-generation mode as UnifiedQA.", "extra_scores": {}, "extra_text": {}}, {"review_id": "jfAP-HzMWqV", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors propose a new test to measure a text model’s multitask accuracy based on 57 different tasks in Humanities, Social Science and STEM subjects. The authors experiment with the GPT-3 models of various sizes and UnifiedQA, and the results suggest that the size of the model may be one of the important factors in achieving higher performance on all the tasks.\n\nThis is an interesting work that tries to tackle a very important problem of how successful are large language models across multiple tasks. It would help the paper to have a more thorough discussion of the results. Based on the reported results, larger GPT-3 models perform better but a smaller UnifiedQA model outperforms GPT-2 by a substantial margin, why is this? \n\nIn addition, it would be interesting to see a comparison of GPT to BERT and XLNet. Such a comparison would first of all show if the idea generalizes to other types of language models. Second, it would emphasize the advantages/disadvantages of autoencoder-based vs autoregressive models and could potentially provide additional insights on how important these differences are for different tasks.\n\nGiven the authors provide additional discussion, I would like to see this paper and the associated dataset to be presented at the conference.", "extra_scores": {}, "extra_text": {}}, {"review_id": "C5ct_Jmep2T", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper focuses on coming up with 57 different tasks and measure the performance of these large scale transformer models such as GPT3 on these different tasks. The main claims of this paper are to demonstrate these large-scale models still struggle to use the knowledge it has learned during the pretraining phase and these models struggle to on calculation-intensive tasks. Further one of the more important contributions of this work includes the massive multi-task dataset that comprises 57 different subjects.\n\n1. This study shows how far off these language models are when compared to how humans use knowledge and commonsense reasoning to solve tasks. Along with that the collecting a comprehensive multi-task dataset that covers the depth and breadth of the multiple subjects would help the community understand these models better. \n2.  One important comparison that would have definitely helped is trying to understand the gaps in knowledge between a GPT-3 to Unified QA. The difference in performance across the tasks seems high especially comparing an 11B to 175B.\n\nQuestions:\n1. How would someone reproduce these experiments considering that widespread access GPT-3 models are not available to the general research community? \n2. How was the level of difficulty of a subject measured? Was it already available when the datasets were being obtained?\n3. Can you elaborate on the results from table 1 which shows that an 11B UnifiedQA model outperforms 175B GPT-3 model? Are these statistical significant?\n\n\n===================== After reading the rebuttal ========\nI thank the authors for providing further information and answering the question raised by the reviewers. Based on the responses and clarifying the issues I had, I have adjusted my score accordingly and improved it from a 5 to 6. \n\n\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an extensive evaluation of two language models: GPT-3 and UnifiedQA on 57 tasks. The results demonstrate that these models are still far from expert-level accuracy and do not know when they are wrong.

I think this is an interesting paper that provides useful insights into the capability of large-scale language models. The authors also plan to release their dataset and have addressed some of the concerns from the reviewers to improve the paper during the rebuttal period.
Decision: Accept (Poster)