Measuring Massive Multitask Language Understanding

1 INTRODUCTION . Natural Language Processing ( NLP ) models have achieved superhuman performance on a number of recently proposed benchmarks . However , these models are still well below human level performance for language understanding as a whole , suggesting a disconnect between our benchmarks and the actual capabilities of these models . The General Language Understanding Evaluation benchmark ( GLUE ) ( Wang et al. , 2018 ) was introduced in 2018 to evaluate performance on a wide range of NLP tasks , and top models achieved superhuman performance within a year . To address the shortcomings of GLUE , researchers designed the SuperGLUE benchmark with more difficult tasks ( Wang et al. , 2019 ) . About a year since the release of SuperGLUE , performance is again essentially human-level ( Raffel et al. , 2019 ) . While these benchmarks evaluate linguistic skills more than overall language understanding , an array of commonsense benchmarks have been proposed to measure basic reasoning and everyday knowledge ( Zellers et al. , 2019 ; Huang et al. , 2019 ; Bisk et al. , 2019 ) . However , these recent benchmarks have similarly seen rapid progress ( Khashabi et al. , 2020 ) . Overall , the near human-level performance on these benchmarks suggests that they are not capturing important facets of language understanding . Transformer models have driven this recent progress by pretraining on massive text corpora , including all of Wikipedia , thousands of books , and numerous websites . These models consequently see extensive information about specialized topics , most of which is not assessed by existing NLP benchmarks . It consequently remains an open question just how capable current language models are at learning and applying knowledge from many domains . To bridge the gap between the wide-ranging knowledge that models see during pretraining and the existing measures of success , we introduce a new benchmark for assessing models across a diverse set of subjects that humans learn . We design the benchmark to measure knowledge acquired during pretraining by evaluating models exclusively in zero-shot and few-shot settings . This makes the benchmark more challenging and more similar to how we evaluate humans . The benchmark covers 57 subjects across STEM , the humanities , the social sciences , and more . It ranges in difficulty from an elementary level to an advanced professional level , and it tests both world knowledge and problem solving ability . Subjects range from traditional areas , such as mathematics and history , to more Few Shot Prompt and Predicted Answer How many numbers are in the list 25 , 26 , ... , 100 ? . ( A ) 75 ( B ) 76 ( C ) 22 ( D ) 23 Answer : B. Compute i + i2 + i3 + ··· + i 258 + i259 . ( A ) -1 ( B ) 1 ( C ) i ( D ) -i Answer : A . If 4 daps = 7 yaps , and 5 yaps = 3 baps , how many daps equal 42 baps ? ( A ) 28 ( B ) 21 ( C ) 40 ( D ) 30 Answer : C␣ The following are multiple choice questions about high school mathematics . ( a ) An example of few-shot learning and inference using GPT-3 . The blue underlined bold text is the autocompleted response from GPT-3 , while the preceding text is the user-inputted prompt . In this 2-shot learning example , there are two instruction examples and one initially incomplete example . On average , GPT-3 has low accuracy on high school mathematics questions . Small Medium Large X-Large Model Size 20 30 40 50 60 70 80 90 Pe rfo rm an ce ( % ) GPT-3 Few Shot Test Performance Commonsense Linguistics Knowledge ( Ours ) ( b ) Performance on a commonsense benchmark ( HellaSwag ) , a linguistic understanding benchmark ( SuperGLUE ) , and the massive multitask test . On previous benchmarks , smaller models start well above random chance levels and exhibit more continuous improvements with model size increases , but on our test , GPT-3 moves beyond random chance with the largest model . specialized areas like law and ethics ( Hendrycks et al. , 2020 ) . The granularity and breadth of the subjects makes the benchmark ideal for identifying a model ’ s blind spots . We find that meaningful progress on our benchmark has only become possible in recent months . In particular , few-shot models up to 13 billion parameters ( Brown et al. , 2020 ) achieve random chance performance of 25 % accuracy , but the 175 billion parameter GPT-3 model reaches a much higher 43.9 % accuracy ( see Figure 1b ) . On the other hand , unlike human professionals GPT-3 does not excel at any single subject . Instead , we find that performance is lopsided , with GPT-3 having almost 70 % accuracy for its best subject but near-random performance for several other subjects . Our results indicate that while recent advances have been impressive , state-of-the-art models still struggle at learning and applying knowledge from pretraining . The tasks with near-random accuracy include calculation-heavy subjects such as physics and mathematics and subjects related to human values such as law and morality . This second weakness is particularly concerning because it will be important for future models to have a strong understanding of what is legal and what is ethical . Worryingly , we also find that GPT-3 does not have an accurate sense of what it does or does not know since its average confidence can be up to 24 % off from its actual accuracy . We comprehensively evaluate the breadth and depth of a model ’ s text understanding by covering numerous topics that humans are incentivized to learn . Since our test consists in 57 tasks , it can be used to analyze aggregate properties of models across tasks and to track important shortcomings . The test and code is available at github.com/hendrycks/test . 2 RELATED WORK . Pretraining . The dominant paradigm in NLP is to pretrain large models on massive text corpora including educational books and websites . In the process , these models are exposed to information about a wide range of topics . Petroni et al . ( 2019 ) found that recent models learn enough information from pretraining that they can serve as knowledge bases . However , no prior work has comprehensively measured the knowledge models have across many real-world domains . Until recently , researchers primarily used fine-tuned models on downstream tasks ( Devlin et al. , 2019 ) . However , larger pretrained models like GPT-3 ( Brown et al. , 2020 ) have made it possible to achieve competitive performance without fine-tuning by using few-shot learning , which removes the need for a large fine-tuning set . With the advent of strong zero-shot and few-shot learning , it is now possible to curate a diverse set of tasks for evaluation and remove the possibility of models on “ spurious cues ” ( Geirhos et al. , 2020 ; Hendrycks et al. , 2019b ) in a dataset to achieve high performance . Benchmarks . Many recent benchmarks aim to assess a model ’ s general world knowledge and basic reasoning ability by testing its “ commonsense. ” A number of commonsense benchmarks have been proposed in the past year , but recent models are already nearing human-level performance on several of these , including HellaSwag ( Zellers et al. , 2019 ) , Physical IQA ( Bisk et al. , 2019 ) , and CosmosQA ( Huang et al. , 2019 ) . By design , these datasets assess abilities that almost every child has . In contrast , we include harder specialized subjects that people must study to learn . Some researchers have suggested that the future of NLP evaluation should focus on Natural Language Generation ( NLG ) ( Zellers et al. , 2020 ) , an idea that reaches back to the Turing Test ( Turing , 1950 ) . However , NLG is notoriously difficult to evaluate and lacks a standard metric ( Sai et al. , 2020 ) . Consequently , we instead create a simple-to-evaluate test that measures classification accuracy on multiple choice questions . While several question answering benchmarks exist , they are comparatively limited in scope . Most either cover easy topics like grade school subjects for which models can already achieve strong performance ( Clark et al. , 2018 ; Khot et al. , 2019 ; Mihaylov et al. , 2018 ; Clark et al. , 2019 ) , or are focused on linguistic understanding in the form of reading comprehension ( Lai et al. , 2017 ; Richardson et al. , 2013 ) . In contrast , we include a wide range of difficult subjects that go far beyond linguistic understanding . 3 A MULTITASK TEST . We create a massive multitask test consisting of multiple-choice questions from various branches of knowledge . The test spans subjects in the humanities , social sciences , hard sciences , and other areas that are important for some people to learn . There are 57 tasks in total , which is also the number of Atari games ( Bellemare et al. , 2013 ) , all of which are listed in Appendix B . The questions in the dataset were manually collected by graduate and undergraduate students from freely available sources online . These include practice questions for tests such as the Graduate Record Examination and the United States Medical Licensing Examination . It also includes questions designed for undergraduate courses and questions designed for readers of Oxford University Press books . Some tasks cover a subject , like psychology , but at a specific level of difficulty , such as “ Elementary , ” “ High School , ” “ College , ” or “ Professional. ” For example , the “ Professional Psychology ” task draws on questions from freely available practice questions for the Examination for Professional Practice in Psychology , while the “ High School Psychology ” task has questions like those from Advanced Placement Psychology examinations . We collected 15908 questions in total , which we split into a few-shot development set , a validation set , and a test set . The few-shot development set has 5 questions per subject , the validation set may be used for selecting hyperparameters and is made of 1540 questions , and the test set has 14079 questions . Each subject contains 100 test examples at the minimum , which is longer than most exams designed to assess people . Human-level accuracy on this test varies . Unspecialized humans from Amazon Mechanical Turk obtain 34.5 % accuracy on this test . Meanwhile , expert-level performance can be far higher . For example , real-world test-taker human accuracy at the 95th percentile is around 87 % for US Medical Licensing Examinations , and these questions make up our “ Professional Medicine ” task . If we take the 95th percentile human test-taker accuracy for exams that build up our test , and if we make an educated guess when such information is unavailable , we then estimate that expert-level accuracy is approximately 89.8 % . Since our test aggregates different subjects and several levels of difficulty , we measure more than straightforward commonsense or narrow linguistic understanding . Instead , we measure arbitrary When you drop a ball from rest it accelerates downward at 9.8 m/s² . If you instead throw it downward assuming no air resistance its acceleration immediately after leaving your hand is ( A ) 9.8 m/s² ( B ) more than 9.8 m/s² ( C ) less than 9.8 m/s² ( D ) Can not say unless the speed of throw is given . C on ce pt ua l Ph ys ic s C ol le ge M at he m at ic s In the complex z-plane , the set of points satisfying the equation z² = |z|² is a ( A ) pair of points ( B ) circle ( C ) half-line ( D ) line Figure 4 : Examples from the Conceptual Physics and College Mathematics STEM tasks . real-world text understanding . Since models are pretrained on the Internet , this enables us to test how well they can extract useful knowledge from massive corpora . Future models that use this test could be single models or a mixture of experts model . To succeed at our test , future models should be well-rounded , possess extensive world knowledge , and develop expert-level problem solving ability . These properties make the test likely to be an enduring and informative goalpost .

Summary: The paper proposes a benchmark for NLP models. The purpose of this test is to measure the model's knowledge in 57 topics covered by approx 15000 tasks in total, each formulated as a closed-form question in zero-shot and few-shot settings. Most of the tasks were taken from different human examination sets. Then, the paper provides results of experiments with the latest (GPT-3 and T5 based) models along with some quantitative and qualitative observations.