Measuring Massive Multitask Language Understanding

1 INTRODUCTION . Natural Language Processing ( NLP ) models have achieved superhuman performance on a number of recently proposed benchmarks . However , these models are still well below human level performance for language understanding as a whole , suggesting a disconnect between our benchmarks and the actual capabilities of these models . The General Language Understanding Evaluation benchmark ( GLUE ) ( Wang et al. , 2018 ) was introduced in 2018 to evaluate performance on a wide range of NLP tasks , and top models achieved superhuman performance within a year . To address the shortcomings of GLUE , researchers designed the SuperGLUE benchmark with more difficult tasks ( Wang et al. , 2019 ) . About a year since the release of SuperGLUE , performance is again essentially human-level ( Raffel et al. , 2019 ) . While these benchmarks evaluate linguistic skills more than overall language understanding , an array of commonsense benchmarks have been proposed to measure basic reasoning and everyday knowledge ( Zellers et al. , 2019 ; Huang et al. , 2019 ; Bisk et al. , 2019 ) . However , these recent benchmarks have similarly seen rapid progress ( Khashabi et al. , 2020 ) . Overall , the near human-level performance on these benchmarks suggests that they are not capturing important facets of language understanding . Transformer models have driven this recent progress by pretraining on massive text corpora , including all of Wikipedia , thousands of books , and numerous websites . These models consequently see extensive information about specialized topics , most of which is not assessed by existing NLP benchmarks . It consequently remains an open question just how capable current language models are at learning and applying knowledge from many domains . To bridge the gap between the wide-ranging knowledge that models see during pretraining and the existing measures of success , we introduce a new benchmark for assessing models across a diverse set of subjects that humans learn . We design the benchmark to measure knowledge acquired during pretraining by evaluating models exclusively in zero-shot and few-shot settings . This makes the benchmark more challenging and more similar to how we evaluate humans . The benchmark covers 57 subjects across STEM , the humanities , the social sciences , and more . It ranges in difficulty from an elementary level to an advanced professional level , and it tests both world knowledge and problem solving ability . Subjects range from traditional areas , such as mathematics and history , to more Few Shot Prompt and Predicted Answer How many numbers are in the list 25 , 26 , ... , 100 ? . ( A ) 75 ( B ) 76 ( C ) 22 ( D ) 23 Answer : B. Compute i + i2 + i3 + ··· + i 258 + i259 . ( A ) -1 ( B ) 1 ( C ) i ( D ) -i Answer : A . If 4 daps = 7 yaps , and 5 yaps = 3 baps , how many daps equal 42 baps ? ( A ) 28 ( B ) 21 ( C ) 40 ( D ) 30 Answer : C␣ The following are multiple choice questions about high school mathematics . ( a ) An example of few-shot learning and inference using GPT-3 . The blue underlined bold text is the autocompleted response from GPT-3 , while the preceding text is the user-inputted prompt . In this 2-shot learning example , there are two instruction examples and one initially incomplete example . On average , GPT-3 has low accuracy on high school mathematics questions . Small Medium Large X-Large Model Size 20 30 40 50 60 70 80 90 Pe rfo rm an ce ( % ) GPT-3 Few Shot Test Performance Commonsense Linguistics Knowledge ( Ours ) ( b ) Performance on a commonsense benchmark ( HellaSwag ) , a linguistic understanding benchmark ( SuperGLUE ) , and the massive multitask test . On previous benchmarks , smaller models start well above random chance levels and exhibit more continuous improvements with model size increases , but on our test , GPT-3 moves beyond random chance with the largest model . specialized areas like law and ethics ( Hendrycks et al. , 2020 ) . The granularity and breadth of the subjects makes the benchmark ideal for identifying a model ’ s blind spots . We find that meaningful progress on our benchmark has only become possible in recent months . In particular , few-shot models up to 13 billion parameters ( Brown et al. , 2020 ) achieve random chance performance of 25 % accuracy , but the 175 billion parameter GPT-3 model reaches a much higher 43.9 % accuracy ( see Figure 1b ) . On the other hand , unlike human professionals GPT-3 does not excel at any single subject . Instead , we find that performance is lopsided , with GPT-3 having almost 70 % accuracy for its best subject but near-random performance for several other subjects . Our results indicate that while recent advances have been impressive , state-of-the-art models still struggle at learning and applying knowledge from pretraining . The tasks with near-random accuracy include calculation-heavy subjects such as physics and mathematics and subjects related to human values such as law and morality . This second weakness is particularly concerning because it will be important for future models to have a strong understanding of what is legal and what is ethical . Worryingly , we also find that GPT-3 does not have an accurate sense of what it does or does not know since its average confidence can be up to 24 % off from its actual accuracy . We comprehensively evaluate the breadth and depth of a model ’ s text understanding by covering numerous topics that humans are incentivized to learn . Since our test consists in 57 tasks , it can be used to analyze aggregate properties of models across tasks and to track important shortcomings . The test and code is available at github.com/hendrycks/test . 2 RELATED WORK . Pretraining . The dominant paradigm in NLP is to pretrain large models on massive text corpora including educational books and websites . In the process , these models are exposed to information about a wide range of topics . Petroni et al . ( 2019 ) found that recent models learn enough information from pretraining that they can serve as knowledge bases . However , no prior work has comprehensively measured the knowledge models have across many real-world domains . Until recently , researchers primarily used fine-tuned models on downstream tasks ( Devlin et al. , 2019 ) . However , larger pretrained models like GPT-3 ( Brown et al. , 2020 ) have made it possible to achieve competitive performance without fine-tuning by using few-shot learning , which removes the need for a large fine-tuning set . With the advent of strong zero-shot and few-shot learning , it is now possible to curate a diverse set of tasks for evaluation and remove the possibility of models on “ spurious cues ” ( Geirhos et al. , 2020 ; Hendrycks et al. , 2019b ) in a dataset to achieve high performance . Benchmarks . Many recent benchmarks aim to assess a model ’ s general world knowledge and basic reasoning ability by testing its “ commonsense. ” A number of commonsense benchmarks have been proposed in the past year , but recent models are already nearing human-level performance on several of these , including HellaSwag ( Zellers et al. , 2019 ) , Physical IQA ( Bisk et al. , 2019 ) , and CosmosQA ( Huang et al. , 2019 ) . By design , these datasets assess abilities that almost every child has . In contrast , we include harder specialized subjects that people must study to learn . Some researchers have suggested that the future of NLP evaluation should focus on Natural Language Generation ( NLG ) ( Zellers et al. , 2020 ) , an idea that reaches back to the Turing Test ( Turing , 1950 ) . However , NLG is notoriously difficult to evaluate and lacks a standard metric ( Sai et al. , 2020 ) . Consequently , we instead create a simple-to-evaluate test that measures classification accuracy on multiple choice questions . While several question answering benchmarks exist , they are comparatively limited in scope . Most either cover easy topics like grade school subjects for which models can already achieve strong performance ( Clark et al. , 2018 ; Khot et al. , 2019 ; Mihaylov et al. , 2018 ; Clark et al. , 2019 ) , or are focused on linguistic understanding in the form of reading comprehension ( Lai et al. , 2017 ; Richardson et al. , 2013 ) . In contrast , we include a wide range of difficult subjects that go far beyond linguistic understanding . 3 A MULTITASK TEST . We create a massive multitask test consisting of multiple-choice questions from various branches of knowledge . The test spans subjects in the humanities , social sciences , hard sciences , and other areas that are important for some people to learn . There are 57 tasks in total , which is also the number of Atari games ( Bellemare et al. , 2013 ) , all of which are listed in Appendix B . The questions in the dataset were manually collected by graduate and undergraduate students from freely available sources online . These include practice questions for tests such as the Graduate Record Examination and the United States Medical Licensing Examination . It also includes questions designed for undergraduate courses and questions designed for readers of Oxford University Press books . Some tasks cover a subject , like psychology , but at a specific level of difficulty , such as “ Elementary , ” “ High School , ” “ College , ” or “ Professional. ” For example , the “ Professional Psychology ” task draws on questions from freely available practice questions for the Examination for Professional Practice in Psychology , while the “ High School Psychology ” task has questions like those from Advanced Placement Psychology examinations . We collected 15908 questions in total , which we split into a few-shot development set , a validation set , and a test set . The few-shot development set has 5 questions per subject , the validation set may be used for selecting hyperparameters and is made of 1540 questions , and the test set has 14079 questions . Each subject contains 100 test examples at the minimum , which is longer than most exams designed to assess people . Human-level accuracy on this test varies . Unspecialized humans from Amazon Mechanical Turk obtain 34.5 % accuracy on this test . Meanwhile , expert-level performance can be far higher . For example , real-world test-taker human accuracy at the 95th percentile is around 87 % for US Medical Licensing Examinations , and these questions make up our “ Professional Medicine ” task . If we take the 95th percentile human test-taker accuracy for exams that build up our test , and if we make an educated guess when such information is unavailable , we then estimate that expert-level accuracy is approximately 89.8 % . Since our test aggregates different subjects and several levels of difficulty , we measure more than straightforward commonsense or narrow linguistic understanding . Instead , we measure arbitrary When you drop a ball from rest it accelerates downward at 9.8 m/s² . If you instead throw it downward assuming no air resistance its acceleration immediately after leaving your hand is ( A ) 9.8 m/s² ( B ) more than 9.8 m/s² ( C ) less than 9.8 m/s² ( D ) Can not say unless the speed of throw is given . C on ce pt ua l Ph ys ic s C ol le ge M at he m at ic s In the complex z-plane , the set of points satisfying the equation z² = |z|² is a ( A ) pair of points ( B ) circle ( C ) half-line ( D ) line Figure 4 : Examples from the Conceptual Physics and College Mathematics STEM tasks . real-world text understanding . Since models are pretrained on the Internet , this enables us to test how well they can extract useful knowledge from massive corpora . Future models that use this test could be single models or a mixture of experts model . To succeed at our test , future models should be well-rounded , possess extensive world knowledge , and develop expert-level problem solving ability . These properties make the test likely to be an enduring and informative goalpost .

Summary: This paper describes a dataset consisting of ~14k multiple-choice questions drawn from many different fields across the humanities and science as well as professional disciplines such as law and medicine. It presents results for GPT-3 models (LMs trained on text corpora with document context) of different scales, as well as for the UnifiedQA model (seq2seq model trained on various QA datasets). Performance of these models is well below their performance on other benchmarks: not above chance for the smaller GPT-3 models, and under 50% average accuracy for the best models.