GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

1 INTRODUCTION . The human ability to understand language is general , flexible , and robust . In contrast , most NLU models above the word level are designed for a specific task and struggle with out-of-domain data . If we aspire to develop models with understanding beyond the detection of superficial correspondences between inputs and outputs , then it is critical to develop a more unified model that can learn to execute a range of different linguistic tasks in different domains . To facilitate research in this direction , we present the General Language Understanding Evaluation ( GLUE ) benchmark : a collection of NLU tasks including question answering , sentiment analysis , and textual entailment , and an associated online platform for model evaluation , comparison , and analysis . GLUE does not place any constraints on model architecture beyond the ability to process single-sentence and sentence-pair inputs and to make corresponding predictions . For some GLUE tasks , training data is plentiful , but for others it is limited or fails to match the genre of the test set . GLUE therefore favors models that can learn to represent linguistic knowledge in a way that facilitates sample-efficient learning and effective knowledge-transfer across tasks . None of the datasets in GLUE were created from scratch for the benchmark ; we rely on preexisting datasets because they have been implicitly agreed upon by the NLP community as challenging and interesting . Four of the datasets feature privately-held test data , which will be used to ensure that the benchmark is used fairly.1 To understand the types of knowledge learned by models and to encourage linguistic-meaningful solution strategies , GLUE also includes a set of hand-crafted analysis examples for probing trained models . This dataset is designed to highlight common challenges , such as the use of world knowledge and logical operators , that we expect models must handle to robustly solve the tasks . 1To evaluate on the private test data , users of the benchmark must submit to gluebenchmark.com To better understand the challenged posed by GLUE , we conduct experiments with simple baselines and state-of-the-art sentence representation models . We find that unified multi-task trained models slightly outperform comparable models trained on each task separately . Our best multi-task model makes use of ELMo ( Peters et al. , 2018 ) , a recently proposed pre-training technique . However , this model still achieves a fairly low absolute score . Analysis with our diagnostic dataset reveals that our baseline models deal well with strong lexical signals but struggle with deeper logical structure . In summary , we offer : ( i ) A suite of nine sentence or sentence-pair NLU tasks , built on established annotated datasets and selected to cover a diverse range of text genres , dataset sizes , and degrees of difficulty . ( ii ) An online evaluation platform and leaderboard , based primarily on privately-held test data . The platform is model-agnostic , and can evaluate any method capable of producing results on all nine tasks . ( iii ) An expert-constructed diagnostic evaluation dataset . ( iv ) Baseline results for several major existing approaches to sentence representation learning . 2 RELATED WORK . Collobert et al . ( 2011 ) used a multi-task model with a shared sentence understanding component to jointly learn POS tagging , chunking , named entity recognition , and semantic role labeling . More recent work has explored using labels from core NLP tasks to supervise training of lower levels of deep neural networks ( Søgaard & Goldberg , 2016 ; Hashimoto et al. , 2017 ) and automatically learning cross-task sharing mechanisms for multi-task learning ( Ruder et al. , 2017 ) . Beyond multi-task learning , much work in developing general NLU systems has focused on sentence-to-vector encoders ( Le & Mikolov , 2014 ; Kiros et al. , 2015 , i.a . ) , leveraging unlabeled data ( Hill et al. , 2016 ; Peters et al. , 2018 ) , labeled data ( Conneau & Kiela , 2018 ; McCann et al. , 2017 ) , and combinations of these ( Collobert et al. , 2011 ; Subramanian et al. , 2018 ) . In this line of work , a standard evaluation practice has emerged , recently codified as SentEval ( Conneau et al. , 2017 ; Conneau & Kiela , 2018 ) . Like GLUE , SentEval relies on a set of existing classification tasks involving either one or two sentences as inputs . Unlike GLUE , SentEval only evaluates sentenceto-vector encoders , making it well-suited for evaluating models on tasks involving sentences in isolation . However , cross-sentence contextualization and alignment are instrumental in achieving state-of-the-art performance on tasks such as machine translation ( Bahdanau et al. , 2015 ; Vaswani et al. , 2017 ) , question answering ( Seo et al. , 2017 ) , and natural language inference ( Rocktäschel et al. , 2016 ) . GLUE is designed to facilitate the development of these methods : It is model-agnostic , allowing for any kind of representation or contextualization , including models that use no explicit vector or symbolic representations for sentences whatsoever . GLUE also diverges from SentEval in the selection of evaluation tasks that are included in the suite . Many of the SentEval tasks are closely related to sentiment analysis , such as MR ( Pang & Lee , 2005 ) , SST ( Socher et al. , 2013 ) , CR ( Hu & Liu , 2004 ) , and SUBJ ( Pang & Lee , 2004 ) . Other tasks are so close to being solved that evaluation on them is relatively uninformative , such as MPQA ( Wiebe et al. , 2005 ) and TREC question classification ( Voorhees et al. , 1999 ) . In GLUE , we attempt to construct a benchmark that is both diverse and difficult . McCann et al . ( 2018 ) introduce decaNLP , which also scores NLP systems based on their performance on multiple datasets . Their benchmark recasts the ten evaluation tasks as question answering , converting tasks like summarization and text-to-SQL semantic parsing into question answering using automatic transformations . That benchmark lacks the leaderboard and error analysis toolkit of GLUE , but more importantly , we see it as pursuing a more ambitious but less immediately practical goal : While GLUE rewards methods that yield good performance on a circumscribed set of tasks using methods like those that are currently used for those tasks , their benchmark rewards systems that make progress toward their goal of unifying all of NLU under the rubric of question answering . 3 TASKS . GLUE is centered on nine English sentence understanding tasks , which cover a broad range of domains , data quantities , and difficulties . As the goal of GLUE is to spur development of generalizable NLU systems , we design the benchmark such that good performance should require a model to share substantial knowledge ( e.g. , trained parameters ) across all tasks , while still maintaining some taskspecific components . Though it is possible to train a single model for each task with no pretraining or other outside sources of knowledge and evaluate the resulting set of models on this benchmark , we expect that our inclusion of several data-scarce tasks will ultimately render this approach uncompetitive . We describe the tasks below and in Table 1 . Appendix A includes additional details . Unless otherwise mentioned , tasks are evaluated on accuracy and are balanced across classes . 3.1 SINGLE-SENTENCE TASKS . CoLA The Corpus of Linguistic Acceptability ( Warstadt et al. , 2018 ) consists of English acceptability judgments drawn from books and journal articles on linguistic theory . Each example is a sequence of words annotated with whether it is a grammatical English sentence . Following the authors , we use Matthews correlation coefficient ( Matthews , 1975 ) as the evaluation metric , which evaluates performance on unbalanced binary classification and ranges from -1 to 1 , with 0 being the performance of uninformed guessing . We use the standard test set , for which we obtained private labels from the authors . We report a single performance number on the combination of the in- and out-of-domain sections of the test set . SST-2 The Stanford Sentiment Treebank ( Socher et al. , 2013 ) consists of sentences from movie reviews and human annotations of their sentiment . The task is to predict the sentiment of a given sentence . We use the two-way ( positive/negative ) class split , and use only sentence-level labels . 3.2 SIMILARITY AND PARAPHRASE TASKS . MRPC The Microsoft Research Paraphrase Corpus ( Dolan & Brockett , 2005 ) is a corpus of sentence pairs automatically extracted from online news sources , with human annotations for whether the sentences in the pair are semantically equivalent . Because the classes are imbalanced ( 68 % positive ) , we follow common practice and report both accuracy and F1 score . QQP The Quora Question Pairs2 dataset is a collection of question pairs from the community question-answering website Quora . The task is to determine whether a pair of questions are semantically equivalent . As in MRPC , the class distribution in QQP is unbalanced ( 63 % negative ) , so we report both accuracy and F1 score . We use the standard test set , for which we obtained private labels from the authors . We observe that the test set has a different label distribution than the training set . STS-B The Semantic Textual Similarity Benchmark ( Cer et al. , 2017 ) is a collection of sentence pairs drawn from news headlines , video and image captions , and natural language inference data . 2 data.quora.com/First-Quora-Dataset-Release-Question-Pairs Each pair is human-annotated with a similarity score from 1 to 5 ; the task is to predict these scores . Follow common practice , we evaluate using Pearson and Spearman correlation coefficients . 3.3 INFERENCE TASKS . MNLI The Multi-Genre Natural Language Inference Corpus ( Williams et al. , 2018 ) is a crowdsourced collection of sentence pairs with textual entailment annotations . Given a premise sentence and a hypothesis sentence , the task is to predict whether the premise entails the hypothesis ( entailment ) , contradicts the hypothesis ( contradiction ) , or neither ( neutral ) . The premise sentences are gathered from ten different sources , including transcribed speech , fiction , and government reports . We use the standard test set , for which we obtained private labels from the authors , and evaluate on both the matched ( in-domain ) and mismatched ( cross-domain ) sections . We also use and recommend the SNLI corpus ( Bowman et al. , 2015 ) as 550k examples of auxiliary training data . QNLI The Stanford Question Answering Dataset ( Rajpurkar et al . 2016 ) is a question-answering dataset consisting of question-paragraph pairs , where one of the sentences in the paragraph ( drawn from Wikipedia ) contains the answer to the corresponding question ( written by an annotator ) . We convert the task into sentence pair classification by forming a pair between each question and each sentence in the corresponding context , and filtering out pairs with low lexical overlap between the question and the context sentence . The task is to determine whether the context sentence contains the answer to the question . This modified version of the original task removes the requirement that the model select the exact answer , but also removes the simplifying assumptions that the answer is always present in the input and that lexical overlap is a reliable cue . This process of recasting existing datasets into NLI is similar to methods introduced in White et al . ( 2017 ) and expanded upon in Demszky et al . ( 2018 ) . We call the converted dataset QNLI ( Question-answering NLI ) .3 RTE The Recognizing Textual Entailment ( RTE ) datasets come from a series of annual textual entailment challenges . We combine the data from RTE1 ( Dagan et al. , 2006 ) , RTE2 ( Bar Haim et al. , 2006 ) , RTE3 ( Giampiccolo et al. , 2007 ) , and RTE5 ( Bentivogli et al. , 2009 ) .4 Examples are constructed based on news and Wikipedia text . We convert all datasets to a two-class split , where for three-class datasets we collapse neutral and contradiction into not entailment , for consistency . WNLI The Winograd Schema Challenge ( Levesque et al. , 2011 ) is a reading comprehension task in which a system must read a sentence with a pronoun and select the referent of that pronoun from a list of choices . The examples are manually constructed to foil simple statistical methods : Each one is contingent on contextual information provided by a single word or phrase in the sentence . To convert the problem into sentence pair classification , we construct sentence pairs by replacing the ambiguous pronoun with each possible referent . The task is to predict if the sentence with the 3An earlier release of QNLI had an artifact where the task could be modeled and solved as an easier task than we describe here . We have since released an updated version of QNLI that removes this possibility . 4RTE4 is not publicly available , while RTE6 and RTE7 do not fit the standard NLI task .

Summary: This paper introduces the General Language Understanding Evaluation (GLUE) benchmark and platform, which aims to evaluate representations of language with an emphasis on generalizability. This is a timely contribution and GLUE will be an impactful resource for the NLP community. This is mitigated, perhaps, somewhat by the recent release of decaNLP. But, as discussed the authors, this has a different focus (re-framing all tasks as QQ) and further does not feature the practical tools released here (leaderboard, error analysis) that will help drive progress.