Discourse-Based Evaluation of Language Understanding

1 INTRODUCTION . Over the last year , novel models for natural language understanding ( NLU ) have made a remarkable amount of progress on a number of widely accepted evaluation benchmarks . The GLUE benchmark ( Wang et al. , 2018 ) , for example , was designed to be a set of challenging NLU tasks , such as question answering , sentiment analysis , and textual entailment ; yet , current state of the art systems surpass human performance estimates on the average score of its subtasks ( Yang et al. , 2019 ) . Similarly , the NLU subtasks that are part of the SentEval framework , a widely used benchmark for the evaluation of sentence-to-vector encoders , are successfully dealt with by current neural models , with scores that exceed the 90 % mark.1 The results on these benchmarks are impressive , but sometimes lead to excessive optimism regarding the ability of current NLU models . For example , based on the resulting performance on the abovementioned benchmarks , a considerable number of researchers has even put forward the claim that their models induce universal representations ( Cer et al. , 2018 ; Kiros & Chan , 2018 ; Subramanian et al. , 2018 ; Wieting et al. , 2015 ; Liu et al. , 2019 ) . It is important to note , however , that benchmarks like SentEval and GLUE are primarily focusing on semantic aspects , i.e . the literal and uncontextualized content of text . While the semantics of language is without doubt an important aspect of language , we believe that a single focus on semantic aspects leads to an impoverished model of language . For a versatile model of language , other aspects of language , viz . pragmatic aspects , equally need to be taken into account . Pragmatics focuses on the larger context that surrounds a particular textual instance , and they are central to meaning representations that aspire to lay a claim to universality . Consider the following utterance : ( 1 ) You ’ re standing on my foot . The utterance in ( 1 ) has a number of direct implications that are logically entailed by the utterance above , such as the implication that the hearer is standing on a body part of the speaker , and the implication that the speaker is touching the hearer . But there are also more indirect implications , that are not literally expressed , but need to be inferred from the context , such as the implication that the speaker wants the hearer to move away from them . The latter kind of implication , that is indirectly 1http : //nlpprogress.com/english/semantic_textual_similarity.html implied by the context of an utterance , is called implicature—a term coined by Grice ( 1975 ) . In real world applications , recognizing the implicatures of a statement is arguably more important than recognizing its mere semantic content . The implicatures that are conveyed by an utterance are highly dependent on its illocutionary force ( Austin , 1975 ) . In Austin ’ s framework , the locution is the literal meaning of an utterance , while the illocution is the goal that the utterance tries to achieve . When we restrict the meaning of ( 1 ) to its locution , the utterance is reduced to the mere statement that the hearer is standing on the speaker ’ s foot . However , when we also take its illocution into account , it becomes clear that the speaker actually formulates the request that the speaker step away . The utterance ’ s illocution is clearly an important part of the entire meaning of the utterance , that is complementary to the literal content ( Green , 2000 ) .2 The example above makes clear that pragmatics is a fundamental aspect of the meaning of an utterance . Semantics focuses on the literal content of utterances , but not on the kind of goal the speaker is trying to achieve . Pragmatic ( i.e . discourse-based ) tasks focus on the actual use of language , so a discourse-centric evaluation could by construction be a better fit to evaluate how NLU models perform in practical use cases , or at least should be used as a complement to semantics-focused evaluations benchmarks . Ultimately , many use cases of NLP models are related to conversation with end users or analysis of structured documents . In such cases , discourse analysis ( i.e . the ability to parse high-level textual structures that take into account the global context ) is a prerequisite for human level performance . Moreover , standard benchmarks often strongly influence the evolution of NLU models , which means they should be as exhaustive as possible , and closely related to the models ’ end use cases . In this work , we compile a list of 11 discourse-focused tasks that are meant to complement existing benchmarks . We propose : ( i ) A new evaluation benchmark , named DiscEval , which we make publicly available.3 ( ii ) Derivations of human accuracy estimates for some of the tasks . ( iii ) Evaluation on these tasks of a state of the art generalizable NLU model , viz . BERT , alongside BERT augmented with auxiliary finetunings . ( iv ) New comparisons of discourse-based and Natural Language Inference based training signals showing that the most widely used auxiliary finetuning dataset , viz . MNLI , is not the best performing on DiscEval , which suggests a margin for improvements . 2 RELATED WORK . Evaluation methods of NLU have been the object of heated debates since the proposal of the Turing Test . Automatic evaluations relying on annotated datasets are arguably limited but they became a standard . They can be based on sentence similarity ( Agirre et al. , 2012 ) , leveraging human annotated scores of similarity between sentence pairs . Predicting similarity between two sentences requires some representation of their semantic content beyond their surface form , and sentence similarity estimation tasks can potentially encompass many aspects , but it is not clear how humans annotators weight semantic , stylistic , and discursive aspects while rating . Using a set of more focused and clearly defined tasks has been a popular approach . Kiros et al . ( 2015 ) proposed a set of tasks and tools for sentence understanding evaluation . These 13 tasks were compiled in the SentEval ( Conneau et al. , 2017 ) evaluation suite designed for automatic evaluation of pre-trained sentence embeddings . SentEval tasks are mostly based on sentiment analysis , semantic sentence similarity and natural language inference . Since SentEval evaluates sentence embeddings , the users have to provide a sentence encoder that is not finetuned during the evaluation . GLUE ( Wang et al. , 2018 ) proposes to evaluate language understanding with less constraints than SentEval , allowing users not to rely on explicit sentence embedding based models . They compile 9 classification or regression tasks that are carried out for sentences or sentence pairs . 3 tasks focus on semantic similarity , and 4 tasks are based on NLI , which makes GLUE arguably semantics-based , 2In order to precisely determine their illocution , utterances have been categorized into classes called speech acts ( Searle et al. , 1980 ) , such as ASSERTION , QUESTION or ORDER which have different kinds of effects on the world . For instance , constative speech acts ( e.g . the sky is blue ) describe a state of the world and are either true or false while performative speech acts ( e.g . I declare you husband and wife ) can change the world upon utterance ( Austin , 1975 ) . 3https : //github.com/disceval/DiscEval even though it also includes sentiment ( Socher et al. , 2013 ) and grammaticality ( Warstadt et al. , 2018 ) . NLI can be regarded as a universal framework for evaluation . In the Recast framework ( Poliak et al. , 2018 ) , existing datasets ( e.g . sentiment analysis ) are formulated as NLI tasks . For instance , based on the sentence don ’ t waste your money , annotated as a negative review , they use handcrafted rules to generate the following example : ( PREMISE : When asked about the product , liam said “ don ’ t waste your money ” , HYPOTHESIS : Liam didn ’ t like the product , LABEL : entailment ) . However , the generated datasets prevent the evaluation to measure directly how well a model deals with the semantic phenomena present in the original dataset , since some sentences use artificially generated reported speech . Thus , NLI data could be used to evaluate discourse analysis , but it is not clear how to generate examples that are not overly artificial . Moreover , it is unclear to what extent instances in existing NLI datasets need to deal with pragmatic aspects ( Bowman , 2016 ) . SuperGLUE ( Wang et al. , 2018 ) updates GLUE with six novel tasks that are selected to be even more challenging . Two of them deal with contextualized lexical semantics , two tasks are a form of question answering , and two of them are NLI problems . One of those NLI tasks , CommitmentBank ( de Marneffe et al. , 2019 ) , is the only explicitly discourse-related task . Another effort towards evaluation of general purpose NLP systems is DecaNLP ( McCann et al. , 2018 ) . The 10 tasks of this benchmarks are all framed as question answering . For example , a question answering task is derived from a sentiment analysis task using artificial questions such as Is this sentence positive or negative ? Four of these tasks deal with semantic parsing , and other tasks include NLI and sentiment analysis . Discourse phenomena can be involved in some tasks ( e.g . the summarization task ) although it is hard to assess to what extent . Discourse relation prediction has punctually been used for sentence representation learning evaluation , by Nie et al . ( 2019 ) and Sileo et al . ( 2019 ) , but they all used only one dataset ( viz . the PDTB ( Prasad et al. , 2008 ) ) , which we included in our benchmark . Discourse has also been considered for evaluation in the field of machine translation . Läubli et al . ( 2018 ) showed that neural models achieve superhuman results on sentence-level translations but that current models yield underwhelming results when considering document-level translations , also making a case for discourse-aware evaluations . Other evaluations , such as linguistic probing or GLUE diagnostics ( Conneau et al. , 2018 ; Belinkov & Glass , 2019 ; Wang et al. , 2019b ) , focus on an internal understanding of what is captured by the models ( e.g . syntax , lexical content ) , rather than measuring performance on external tasks ; they provide a complementary viewpoint , but are outside the scope of this work . 3 PROPOSED TASKS . Our goal is to compile a set of diverse discourse-related tasks . We restrict ourselves to classification either of sentences or sentence pairs and only use publicly available datasets that are absent from other benchmarks ( SentEval/GLUE/SuperGLUE ) . The scores in our tasks are not all meant to be compared to previous work , since we alter some datasets to yield more meaningful evaluations ( we perform duplicate removal or class subsampling when mentioned ) . We found these operations necessary in order to leverage the rare classes and yield more meaningful scores . As an illustration , GUM initially consists of more than 99 % of unattached labels , and SwitchBoard contains 80 % of statements . We first present the tasks we selected , also described in table 1 and appendix A , and then propose a rudimentary taxonomy of how they address different aspects of meaning . PDTB The Penn Discourse Tree Bank ( Prasad et al. , 2014 ) contains a collection of fine-grained implicit ( i.e . not signaled by a discourse marker ) relations between sentences from the news domain in the Penn TreeBank 2.0 . We select the level 2 ( called types in PDTB terminology ) relations as categories . STAC ( Strategic Conversation ; Asher et al. , 2016 ) is a corpus of strategic chat conversations manually annotated with negotiation-related information , dialogue acts and discourse structures in the framework of Segmented Discourse Representation Theory ( SDRT ; Asher & Lascarides , 2003 ) . We only consider pairwise relations between all dialog acts , following Badene et al . ( 2019 ) . We remove duplicate pairs and dialogues that only have non-linguistic utterances ( coming from the game server ) . We subsample dialog act pairs with no relation so that they constitute 20 % of each fold . GUM ( Zeldes , 2017 ) is a corpus of multilayer annotations for texts from various domains ; it includes discourse structure annotations according to Rhetorical Structure Theory ( RST ; Mann & Thompson , 1987 ) . Once again , we only consider pairwise interactions between discourse units ( e.g . sentences/clauses ) . We subsample discourse units with no relation so that they constitute 20 % of each document . We split the examples in train/test/dev sets randomly according to the document they belong to . Emergent ( Ferreira & Vlachos , 2016 ) is composed of pairs of assertions and titles of news articles that are against , for , or neutral with respect to the opinion of the assertion . SwitchBoard ( Godfrey et al. , 1992 ) contains textual transcriptions of dialogs about various topics with annotated speech acts . We remove duplicate examples and subsample Statements and Non Statements so that they constitute 20 % of the examples . We use a custom train/validation split ( 90/10 ratio ) since our preprocessing leads to a drastic size reduction of the original development set . The label of a speech act can be dependent on the context ( previous utterances ) , but we discarded it in this work for the sake of simplicity , even though integration of context could improve the scores ( Ribeiro et al. , 2015 ) . MRDA ( Shriberg et al. , 2004 ) contains textual transcription of multi-party real meetings , with speech act annotations . We remove duplicate examples . We use a custom train/validation split ( 90/10 ratio ) since this deduplication leads to a drastic size reduction of the original development set , and we subsample Statement examples so that they constitute 20 % of the dataset . We also discarded the context . Persuasion ( Carlile et al. , 2018 ) is a collection of arguments from student essays annotated with factors of persuasiveness with respect to a claim ; considered factors are the following : Specificity , Eloquence , Relevance and Strength . For each graded target , we cast the ratings into three quantiles and discard the middle quantile . SarcasmV2 ( Oraby et al. , 2016 ) consists of messages from online forums with responses that may or may not be sarcastic according to human annotations . Squinky dataset ( Lahiri , 2015 ) gathers annotations on Formality , Informativeness , and Implicature , where sentences were graded on a scale from 1 to 7 . The Implicature score is defined as the amount of information that is not explicitly expressed in a sentence . For each target , we cast the ratings into three quantiles and discard the middle quantile . Verifiability ( Park & Cardie , 2014 ) is a collection of online user comments annotated as VerifiableExperiential ( verifiable and about writer ’ s experience ) Verifiable-Non-Experiential or Unverifiable . EmoBank ( Buechel & Hahn , 2017 ) aggregates emotion annotations on texts from various domains using the VAD representation format . The authors define Valence as corresponding to the concept of polarity4 , Arousal as degree of calmness or excitement and Dominance as perceived degree of control over a situation . For each target , we cast the ratings into three quantiles and discard the middle quantile . It has been argued by Halliday ( 1985 ) that linguistic phenomena fall into three metafunctions : ideational for semantics , interpersonal for appeals to the hearer/reader , and textual for form-related aspects . This forms the basis of discourse relation types by Hovy & Maier ( 1992 ) , who call them semantic , interpersonal and presentational . DiscEval tasks cut across these categories , because some of the tasks integrate all aspects when they characterize the speech act or discourse relation category associated to a discourse unit ( mostly sentences ) , an utterance or a pair of these . However , most discourse relations involved focus on ideational aspects , which are thus complemented by tasks insisting on more interpersonal aspects ( e.g . using appeal to emotions , or verifiable arguments ) that help realize speech act intentions . Finally , intentions can achieve their goals with varying degrees of success . This leads us to a rudimentary grouping of our tasks : – The speech act classification tasks ( SwitchBoard , MRDA ) deal with the detection of the intention of utterances . They use the same label set ( viz . DASML , Allen & Core , 1997 ) but different domains and annotation guidelines . A discourse relation also characterizes how an utterance contributes to the coherence of a document/conversation ( e.g through elaboration or contrast ) , so this task requires a form of understanding of the use of a sentence , and how a sentence fits with another sentence in a broader discourse . A discourse relation can be seen as a speech act whose definition is tied to a structured context ( Asher & Lascarides , 2003 ) . Here , three tasks ( PDTB , STAC , GUM ) deal with discourse relation prediction with varying domains and formalisms.5 The Stance detection task can be seen as a coarse-grained discourse relation classification . – Detecting emotional content , verifiability , formality , informativeness or sarcasm is necessary in order to figure out in what realm communication is occurring . A statement can be persuasive , yet poorly informative and unverifiable . Emotions ( Dolan , 2002 ) and power perception ( Pfeffer , 1981 ) can have a strong influence on human behavior and text interpretation . Manipulating emotions can be the main purpose of a speech act as well . Sarcasm is another means of communication and sarcasm detection is in itself a straightforward task for the evaluation of pragmatics , since sarcasm is a clear case of literal meaning being different from the intended meaning . – Persuasiveness prediction is a useful tool to assess whether a model can measure how well a sentence can achieve its intended goal . This aspect is orthogonal to the determination of the goal itself , and is arguably equally important .

Summary: This paper presents a new benchmark for natural language understanding called DiscEval. The benchmark focuses on datasets that more directly measure a model's understanding of the discourse structure and relations in the text. The paper makes direct comparisons to GLUE and especially studies the previous claims of MNLI being a generically good pre-training task. They find that MNLI pretraining, using STILTS, does not help BERT's performance on DiscEval.