Discourse-Based Evaluation of Language Understanding
Authors: Damien Sileo
Tim Van-De-Cruys
Camille Pradel
Philippe Muller
Venue:  (2020)
Keywords: Natural Language Understanding
Pragmatics
Discourse
Semantics
Evaluation
BERT
Natural Language Processing

Abstract: New models for natural language understanding have made unusual progress recently, leading to claims of universal text representations. However, current benchmarks are predominantly targeting semantic phenomena;  we make the case that discourse and pragmatics need to take center stage in the evaluation of natural language understanding.
We introduce DiscEval, a new benchmark for the evaluation of natural language understanding, that unites 11 discourse-focused evaluation datasets. 
DiscEval can be used as supplementary training data in a multi-task learning setup, and is publicly available, alongside the code for gathering and preprocessing the datasets.
Using our evaluation suite, we show that natural language inference, a widely used pretraining task, does not result in genuinely universal representations, which opens a new challenge for multi-task learning.

Under review as a conference paper at ICLR 2020
DISCEvAL: DISCOURSE BASED EVALUATION OF NATURAL LANGUAGE UNDERSTANDING
Anonymous authors Paper under double-blind review
ABSTRACT
New models for natural language understanding have made unusual progress recently, leading to claims of universal text representations. However, current benchmarks are predominantly targeting semantic phenomena; we make the case that discourse and pragmatics need to take center stage in the evaluation of nat- ural language understanding. We introduce DiscEval, a new benchmark for the evaluation of natural language understanding, that unites 11 Riscourse-focused evaluation datasets. DiscEval can be used as supplementary training data a multi- task learning setup, and is publicly available, alongside the code for gathering and preprocessing the datasets. Using our evaluation suite, we show that natural language inference, widely used pretraining task, does not result in genuinely universal representations, which opens new challenge for multi-task learning.
INTRODUCTION
Over the last year, novel models for natural language understanding (NLU) have made remarkable amount of progress on a number of widely accepted evaluation benchmarks. The GLUE benchmark (Wang al., 2018), for example, was designed be set of challenging NLU tasks, such question answering, sentiment analysis, and textual entailment; yet, current state of the art systems surpass human performance estimates on the average score of its subtasks (Yang et al.] 2019). Similarly, the NLU subtasks that are part of the SentEval framework, a widely used benchmark for the evaluation of sentence-to- tooeeett encoders, are successfully dealt with by current neural models, with scores that exceed the 90% mark
The results these benchmarks impressive, but sometimes lead to excessive optimism regarding the ability of current NLJU models. For example, based on the resulting performance on the above- mentioned benchmarks, considerable number of researchers has even put forward the claim that their models induce universal representations (Cer et al., [2018] Kiros & Chan 2018; Subramanian et al.)
2018 Wieting et al. 2013 Liu al. 2019). It important note, however, that benchmarks like SentEval and GLUE are primarily focusing on semantic aspects, i.e. the literal and uncontextualized content of text. While the semanties of language without doubt an important aspect of language, we believe that single focus on semantic aspects leads to impoverished model of language.
For versatile model of language, other aspects of language, viz. pragmatic aspects, equally need to be taken into account. Pragmatics focuses on the larger context that surrounds a particular textual instance, and they are central to meaning representations that aspire to lay a claim to universality. Consider the following utterance:
(1) You're 're standing on my foot.
The utterance in (1) has number of direct implications that are logically entailed by the utterance above, such as the implication that the hearer is standing on a body part of the speaker, and the implication that the speaker is touching the hearer. But there are also more indirect implications, that are not literally expressed, but need to be inferred from the context, such as the implication that the speaker wants the hearer move away from them. The latter kind of implication, that is indirectly
http://n1pprogreesee com com/engl sh/ wwwantmmntiiimm weranntc__teee semantie_teextual 1_simi larity. html

---

Under review as a conference paper at ICLR 2020
implied by the context of utterance, is called implicature- term coined by Grice (1755. In real world applications, recognizing the implicatures of a statement is arguably more important than recognizing its mere semantic content.
The implicatures that are conveyed by an utterance are highly dependent on its illocutionary force (Austin 1975). In Austin's framework, the locution is the literal meaning of an utterance, while the illocution is the goal that the utterance tries to achieve. When we restrict the meaning [ to its locution, the utterance is reduced to the mere statement that the hearer is standing on the speaker's foot. However, when we also take its illocution into account, it becomes clear that the speaker actually formulates the request that the speaker step away. The utterance's illocution is clearly important part of the entire meaning of the utterance, that is complementary to the literal content (Green 2000
The example above makes clear that pragmatics is a fundamental aspect of the meaning of an utterance. Semanties focuses on the literal content of utterances, but not on the kind of goal the speaker is trying to achieve. Pragmatic (i.c. discourse-basett tasks focus on the actual use of language, so a discourse-centric evaluation could by construction be a better fit to evaluate how NLU models perform in practical use cases, at least should be used evaluations benchmarks. Ultimately, many use cases of NLP models are related to conversation with end users or analysis of structured documents. In such cases, discourse analysis (i.e. the ability to parse high-level textual structures that take into account the global context) is a prerequisite for human level performance. Moreover, standard benchmarks often strongly influence the evolution of NLU models, which means they should be as exhaustive as possible, and closely related to the models' end use cases.
complement to semantics-focusee
In this work, we compile a list of 11 Aiscourse-focusee tasks that are meant to complement existing benchmarks. We propose: (i) A new evaluation benchmark, named DiscEval, which make publicly available (ii) Derivations of human accuracy estimates for some of the tasks. (iii) Evaluation on these tasks of state of the art generalizable NLU model, viz. BERT, alongside BERT augmented with auxiliary finetunings. (iv) New comparisons of discourre-basse and Natural Language Inference based training signals showing that the most widely used auxiliary finetuning dataset, viz. MNLI, is not the best performing on DiscEval, which suggests a margin for improvements.
2 RELATED WORK
Evaluation methods of NLU have been the object of heated debates since the proposal of the Turing Test. Automatic evaluations relying on annotated datasets are arguably limited but they became a standard. They can be based on sentence similarity (Agirre et al. 2012), leveraging human annotated scores of similarity between sentence pairs. Predicting similarity between two sentences requires some representation of their semantic content beyond their surface form, and sentence similarity estimation tasks can potentially encompass many aspects, but it is not clear how humans annotators weight semantic, stylistic, and discursive aspects while rating.
Using a set of more focused and clearly defined tasks has been a popular approach. Kiros et al. (2015) proposed a set of tasks and tools for sentence understanding evaluation. These 13 tasks were compiled in the SentEval (Conneau et al., (017)) evaluation suite designed for automatic evaluation of pre-trained sentence embeddings. SentEval tasks are mostly based on sentiment analysis, semantic sentence similarity and natural language inference. Since SentEval evaluates sentence embeddings, the users have to provide a sentence encoder that is not finetuned during the evaluation.
GLUE (Wang et al. 2018) proposes to evaluate language understanding with less constraints than SentEval, allowing users not to rely on explicit sentence embedding based models. They compile classification regression tasks that are carried out for sentences or sentence pairs. 3 tasks focus on semantic similarity, and 4 tasks are based on NLI, which makes GLUE arguably semantics-based,
"In order to precisely determine their illocution, utterances have been categorized into classes called speech acts (Searle et al. 1980). such ASSERTION, QUESTION ORDER which have different kinds of effects on the world. For instance, constative speech acts (e.g. the sky is blue) describe state of the world and are either true or false while performative speech acts (e.g. I declare you hushand and wife) can change the world upon utterance (Austin 1975).
https::/ thub. com/di sceval /IDi scEval
2

---

Under review as a conference paper at ICLR 2020
even though it also includes sentiment (Socher et al. 2013) and grammaticality (Warstadt al. 2018
NLI can be regarded as universal framework for evaluation. In the Recast framework (Poliak et al. 2018), existing datasets (e.g. sentiment analysis) are formulated as NI.I tasks. For instance, based on the sentence don't It waste your money, annotated as negative review, they use handcrafted rules to generate the following example: (PREMISE: When asked about the product, liam said "don"t 't waste your money", IIY IYPOTHESSS: Liam didn't It like the product, LABEL: entailment). However, the generated datasets prevent the evaluation to measure directly how well a model deals with the semantic phenomena present in the original dataset, since some sentences use artificially generated reported speech. Thus, NLI data could be used to evaluate discourse analysis, but is not clear how to generate examples that are not overly artificial. Moreover, it is unclear to what extent instances in existing NLJ datasets need to deal with pragmatic aspects (Bowman) 2016).
SuperGLUE (Wang et al. 2018) updates GLUE with six novel tasks that are selected to be even more challenging. Two of them deal with contextualized lexical semantics, two tasks are a form of question answering, and two of them NLI problems. One of those NLI tasks, CommitmentBank (de Marneffe et al. 2019), is the only explicitly discourse-related task.
Another effort towards evaluation of general purpose NLP systems is DecaNLP (McCann et al. 2018). The 10 tasks of this benchmarks are all framed as question answering. For example, a question answering task is derived from sentiment analysis task using artificial questions such as Is this sentence positive or negative? Four of these tasks deal with semantic parsing, and other tasks include NLI and sentiment analysis. Discourse phenomena can be involved in some tasks (c.g. the summarization task) although is hard to assess to what extent.
Discourse relation prediction has punctually been used for sentence representation learning evaluation, by Nie et al. 2019) and [Sileo et al. (2019), but they all used only one dataset (viz. the PDTB (Prasad et al. 2008)), which we included in our benchmark. Discourse has also been considered for evaluation the field of machine translation. Läubli et al. (2018) showed that neural models achieve superhuman results on sentence-leve translations but that current models yield underwhelming results when considering document-leww translations, also making a case for discourse aware evaluations.
Other evaluations, such as linguistic probing or GLUE diagnostics (Conneau et al. 2018 Belinkov & Glass 2019] Wang et al. 2019b), focus on an internal understanding of what is captured by the models (e.g. syntax, lexical content), rather than measuring performance on external tasks; they provide a complementary viewpoint, but are outside the scope this work.
PROPOSED TASKS
Our goal is to compile set of diverse discourse-related tasks. We restrict ourselves to classification either of sentences or sentence pairs and only use publicly available datasets that are absent from other benchmarks SeetEval/G..UE//uppGGLLU
The scores in tasks datasets to yield more meaningful evaluations (we perform duplicate removal or class subsampling when mentioned). We found these operations necessary in order leverage the rare classes and yield more meaningful scores. As an illustration, GUM initially consists of more than 99% of unattached labels, and SwitchBoard contains 80% of statements.
all meant be compared to previous work, since we alter some
We first present the tasks we selected, also described in table and appendix A, and then propose a rudimentary taxonomy of how they address different aspects meaning.
The Penn Discourse Tree Bank (Prasad et al.) 2014)) contains a collection of fine-grained implicit (i.e. not signaled by discourse marker) relations between sentences from the news domain in the Penn TreeBank 2.0. We select the level 2 (called types in PDTB terminology) relations as categories.
PDTB
STAC (Strategic Conversation; Asher et al. 2016) is a corpus of strategic chat conversations
manually annotated with regotiation-related information, dialogue acts and discourse structures in the framework of Segmented Discourse Representation Theory (SDRT; Asher & Lascarides 2003). We only consider pairwise relations between all dialog acts, following Badene et al. (2019). We
3

---

Under review as a conference paper at ICLR 2020
dataset
categories
example
class Nuai
PDTB
discourse relation
"it censorship""'it was outrageous"
conjunction 13k
STAC
discourse relation
"what ?"/"i literally lost"
censsiionansser-pall
11k
GUM
discourse relation
"Do not drink"/ underage in your country"
condition 2k
Emergent
stance
"a meteorite landed in "ucaragua."""smiinn metcorite hits managua" "well. a little different, actually
for 2k
SwitchBoard
speech act
hedge 19k
MRDA
speech
"yeah that that 's that 's what meant." "
acknowledge-answee 14k
Persuasion
E/S/S/R
"Ca-operation is essential for team "'roon''ions hunt a team low specificity 0.6k
SarcasmV2
sarcasm presence
"don't quit your day job"/" was going to sell this joke. [..]]' sarcasm 9k
Squinky
I/F
"boo ya."
uninformative, high implicature, unformal 4k
Verifiabilitty verifiability
"I've been physician for 20 years." "I wanted to be there.."
verifiable-experieine 6k
EmoBank
V/AD
valence, high arousal, low dominance
Table I: DiscEval classification datasets. Nurain is the number of examples in the training set. E/S/S/R denotes Elogueee//trragtt//eciiiiilllle denotes Valence/Arouus//Dominnaa
I/I/F is Infomatio//miiclueeFomiiit V/A/D
remove duplicate pairs and dialogues that only have non- non-linguistic utterances (coming from the game server). We subsample dialog act pairs with no relation that they constitute 20% of each fold.
GUM (Zeldes 2017) is a corpus of multilayer annotations for texts from various domains:
includes discourse structure annotations according to Rhetorical Structure Theory (RST; Mann & Thompson 1987). Once again, we only consider pairwise interactions between discourse units (e.g. sentences/clauses) We subsample discourse units with relation that they constitute 20% of each document. We split the examples in train/test/dev sets randomly according to the document they belong
Emergent Ferreiir Vlachos 2016) composed pairs of assertions and titles of news articles that are against, neutral with respect the opinion of the assertion.
SwitchBoard [Godfrey et al. 1992) contains textual transcriptions of dialogs about various topics with annotated speech acts. We remove duplicate examples and subsample Statements and Non Statements so that they constitute 20% of the examples. We use custom train/validation split (90/10 ratio) since our preprocessing leads to drastic size reduction of the original development set. The label of speech act can be dependent on the context (previous utterances). but we discarded it in this work for the sake of simplicity, even though integration of context could improve the scores (Ribeiro) al. 2015).
MRDA (Shriberg et al. 2004) contains textual transcription of multi-party real meetings, with speech act annotations. We remove duplicate examples. We use custom train/validation split (90/10 ratio) since this deduplication leads to a drastic size reduction of the original development set, and we subsample Statement examples so that they constitute 20% of the dataset. We also discarded the context.
Persuasion (Carlile al. 2018 is collection of arguments from student essays annotated with factors persuasiveness with respect to a claim; considered factors are the following: Specificity, Eloquence, Relevance and Strength. For each graded target, cast the ratings into three quantiles and discard the middle quantile.
Sarcasm V2 (Orab et al. 2016) consists of messages from online forums with responses that may or may not be sarcastic according human annotations.
Squinky dataset Lahini 2015) gathers annotations on Formality, Informativeness, and Implicature, where sentences were graded scale from 1 to 7. The Implicature score is defined as the amount of information that not explicitly expressed in a sentence. For each target, we cast the ratings into three quantiles and discard the middle quantile.
4

---

Under review as a conference paper at ICLR 2020
Verifiability Park & Cardie 2014) is collection of online user comments annotated as Verifiable- Experiential (verifiable and about writer's experience) Verifiable Non-Experiential or Unverifiable.
EmoBank (Bucchel & Hahn 2017) aggregates emotion annotations on texts from various domains using the VAD VAD represettatioon format. The authors define Valence as corresponding to the concept of polarity Arousal as degree of calmness or excitement and Dominance as perceived degree of control over a situation. For each target, we cast the ratings into three quantiles and discard the middle quantile.
It has been argued by Halliday (1985) that linguistic phenomena fall into three metafunctions: ideational for semantics, interpersonal for appeals to the hearer/reader, and textual for form-relate related aspects. This forms the basis of discourse relation types by Hovy & Maier (1992), who call them semantic, interpersonal and presentational. DiscEval tasks cut across these categories, because some of the tasks integrate all aspects when they characterize the speech act or discourse relation category associated to a discourse unit (mostly sentences), an utterance or pair of these. However, most discourse relations involved focus on ideational aspects, which are thus complemented by tasks insisting on more interpersonal aspects (e.g. using appeal to emotions, or verifiable arguments) that help realize speech act intentions. Finally, intentions can achieve their goals with varying degrees of success. This leads us to rudimentary grouping of our tasks:
The speech act classification tasks (SwitchBoard, MRDA) deal with the detection of the intention of utterances. They use the same label set (viz. DASML, Allen & Core 1997) but different domains and annotation guidelines. A discourse relation also characterizes how an utterance contributes to the coherence of documnnt/ccovversation (e.g through elaboration or contrast), so this task requires a form of understanding of the use of a sentence, and how a sentence fits with another sentence in broader discourse. A discourse relation can be seen speech act whose definition is tied to a structured context (Asher & Lascarides) 2003). Here, three tasks (PDTB, STAC, GUM) deal with discourse relation prediction with varying domains and formalisms The Stance detection task can be seen as a coarse grarine discourse relation classification.
Detecting emotional content, verifiability, formality, informativeness or sarcasm is necessary in order to figure out in what realm communication is occurring. A statement can be persuasive, yet poorly informative and unverifiable. Emotions (Dolan 2002) and power perception (Pieffer 1981) can have a strong influence on human behavior and text interpretation. Manipulating emotions can be the main purpose of a speech act as well. Sarcasm another means of communication and sarcasm detection is in itself a straightforward task for the evaluation of pragmatics, since sarcasm is clear case of literal meaning being different from the intended meaning.
Persuasiveness prediction is useful tool to assess whether model can measure how well sentence can achieve its intended goal. This aspect is orthogonal to the determination of the goal itself, and is arguably equally important.
4 EVALUATIONS
4.1
MODELS
Our goal is to assess the performance of popular NLU models and the influence of various training signals on DiscEval scores. We evaluate state of the models and baselines on DiscEval using the Jiant Wang el al. 2019c) framework. Our baselines include average of GloVe (Pennington et al. 2014 embeddings (CBoW) and BiLSTM with GloVe and ELMo (Peters et al.
2018) embeddings.
We also evaluate BERT (Devlin et al. 2019) base uncased models, and perform experiments with Supplementary Training on Intermediate Labeled-data Tasks (STILTT; Phang et al. further pretraining step on a data-rich task before the final fine-tuning evaluation on the target task. STILTs can be combined using multitask learning. We use Jiant default parameters and uniform loss weighting when multitasking (a different task is optimized at each training batch).
2018). STILT is a
We finetune BERT with four of such training signals:
"This is the dimension that is widely used in sentiment analysis.
5 'These formalisms have different assumptions about the nature discourse structure.
httpps / thub. com milp://.w..oommmmmmmmmmmmmmmmmmm /tt/////aanntbblo///555555 c338cc33d117 2at.:e3ddede613333332727700000
3cce2587ea2 1 f887a17/
j jiant/con httm://oonnngeeeaaaae/////titiit
5

---

Under review as a conference paper at ICLR 2020
MNLI WWlliams al. 2018 is a collection of 433k sentence pairs manually annotated with contradiction, entailent, or neutral relations. Phang et al. (2018) showed that finetuning with this dataset leads to accuracy improvement all GLUE tasks except CoLA (Warstadt et al. 2018).
DisSent Nie et al. 2019) consists of 4.7M sentence pairs that are separated by discourse marker (from a list of 15 markers). Prediction of discourse markers based on the context clausev/sentences with which they occur has been used training signal for sentence representation learning. The authors used handcrafted rules for each marker in order to ensure that the markers signal an actual relation. DisSent has underwhelming results on the GLUE tasks as a STILT (Wang et al., 2019a).
Discovery Sileo et al., 2019) is another dataset for discourse marker prediction, composed of 174 discourse markers with 10k usage examples for each marker. Sentence pairs were extracted from web data, and the markers come either from the PDTB or from a heuristic automatic extraction.
DiscEval refers to all DiscEval tasks used in multitask setup; since we uniform loss weighting, we discard Persuasion classes other than Strength (note that the other classes can be considered subfactors for strength) in order to prevent the Persuasion task overwhelm the others.
4.2 HUMAN ACCURACY ESTIMATES
For a more insightful comparison, we propose derivations of human accuracy estimates for the datasets we used.
The authors of Sarcasm (Oraby al. 2016) dataset directly report 80% annotator accuracy compared to the gold standard. Prasad et al. (001)) report 84% annotator agreement for PDTB 2.0, which is a lower bound of accuracy. For GUM (Zeldes 2017). an attachment accuracy of 87 22% and labelling accuracy of 88.56% as compared to the gold standard' after instructor adjudication is reported. We interleaved attachment and labelling in our task. Assuming human annotators never predict the non-attached relation, 69.3% is lower bound for human accuracy. Authors of the Verifiability (Park & Cardie, 2014) dataset report an agreement of 87% given the class distribution, which is lower bound of human accuracy. We estimated human accuracy on EmoBank (Buechel & Hahn 2017) with the intermediate datasets provided by the authors. For each target (V.A,D) compute the average standard deviation, and compute the probability (under normality assumption) of each example rating of falling under the wrong category.
0.73 which yields an agreement
Unlike the GLUE benchmark (Nangia & Bowman 2019), we do not yet provide human accuracy estimates obtained in standardized way. The high number of classes would make that process rather more difficult. But our estimates are still useful even though they should be taken with grain of salt.
4.3 OVERALI RESULTS
Task-wise results are presented in table We report the average scores of 6 runs of STILT and finetuning phases.
DiscEval seem to be challenging even for the BERT base model, which has shown strong performance on GLUE (and vastly outperforms the baselines on our tasks). For many tasks, there a STILT that significantly improves the accuracy of BERT. The gap between human accuracy and BERT is particularly high on implicit discourse relation prediction (PDTB and GUM). This task known to be difficult, and previous work has shown that task dedicated models are not yet on par with human performance (Morey et al. 2017). Pretraining on MNLI does not improve the DvscEval average score for the BERT base model. A lower sarcasm detection score could indicate that BERT+MNLI is more focused on the literal content of statements, even though no STIL improves sarcasm detection, All models score below human accuracies, with the exception of emotion classification (but only for the valence classification subtask).
Table 3
shows aggregate results alongside comparisons with GLUE scores. The best overall unsuper- vised result (GLUE+DiscEvaal average) is achieved with Discovery STILT. Combining Discovery and MNLI yields both a high DiscEval and GLUE score, and also yields a high GLUE diagnostics score. All discourse based STILT improve GLUE score, while MNLI does not improve DvscEval average score. DiscEval tasks based sentence pairs seem account for the variance across STI.Ts..

---

Under review as a conference paper at ICLR 2020
PDTB
STAC
GUM
Emergent SwitchB. MRDA Persuasion Sarcasm Squinky
Verif.
EmoBank
CBoW
27.4
32
20.5
59.7
3.8
0.7
70.6
61.1
75.5
74.0
64.0
BiLSTM
25.9
27.7
18.5
45.6
3.7
0.7
62.6
63.1
72.1
74.0
63.5
BIL
27.5
33.5
18.9
55.2
3.7
0.7
67.4
68.9
82.5
74.0
66.9
Previous work
48.2
73.1
81.1
BERT
48.8
48.2
40.9
79.2
38.8
22.3
74.8
77.1
87.5
86.7
76.2
BERT+MNLI
49.1
49.1
42.8
81.2
22.7
71.7
73.4
86.0
76.3
Disla
49.1
57.1
42.8
80.2
40.3
23.1
76.2
75.0
87.6
85.9
76.0
BiRT+
49.4
49.0
43.9
79.8
39.2
22.0
74.7
74.9
87.5
85.9
76.2
B+DisSent+MNLI
49.6
49.2
44.2
80.9
39.8
22.1
74.0
74.1
87.6
85.6
76.4
BiRTvDisyovery
50.7
49.5
42.7
81.7
22.4
71.6
76.7
88.6
86.3
76.6
B+Diicovery+MNNJ
51.3
49.4
43.1
80.7
40.3
22.2
73.6
75.1
88.9
86.8
76.0
Human estimate
84.0
69.3
80.0
87.0
73.1
Table 2: Transfer test scores across DiscEval tasks; we report the average when the dataset has several classification tasks (as in Squinky, EmoBank and Persuasion); B(ERT)+X refers to BERT pretrained classification model after an auxiliary finetuning phase on task X. All scores are accuracy scores except SwitchBoard/MRDA (which are macro-Fl scores). Previous work refer to the best scores from previous work that used similar setup. PDTB score is from Bai & Zhao (2018). Emergent score is from Ferreira & Vlachos (2016). Verifiability score is derived from Park & Cardie (2014).
MNLI has been suggested good default auxiliary training task based evaluation on GLUE (Phang et al. 2018) and SentEval (Conneau et al. 2017). However, our evaluation suggests that finetuning a model with MNLI alone has significant drawbacks.
More detailed results for datasets with several subtasks are shown in table We note that MNLI STILT significantly decreases relevance estimation performance (on BERT base and while multi- tasking with DisSent). Many models surpass the human estimate at valence prediction, well studied task, but interestingly this the case for Arousal and Dominance prediction.
Div D.E.-Painy Aa.-Pairs D.E.-SingleAve GLUE
GLUEatiagneekses
BERT
61.81.44
$7.91.5
62.31.3
74.711.2
31.7.L.3
BERT+MNLI
61.71.5
57.2.1.5
62.21.4
77.01.2
32,51.6
Bval MTL
63.0
60.0
62.61.2
75.31.22
31.611.3
BERT+DisSent
62.0+.4
58.4+.4
62.2=33
75.1
31.5#.3
B+DisSent-MNLI 62.1
58.2t.4
62.3=.2
76.6
32.4
BERT+Discovery 62.4+.3
58.2+.4
62.7=.3
75.0
31.3
B+Discovery+MNII
62.5+
58.5+
62.8
76.6+.2
33.3+.2
Table Aggregated transfer test accuracies across DiscEval and comparison with GLUE validation downstream and diagnostic tasks (GLUE diagnostic tasks evaluate NLI performance under pres- ence of linguistic phenomena such as negation, quantification, use of common sense); BERT+X refers to BERT pretrained classification model after auxiliary finetuning phase on task X: D.E.- Pairs G is the average of DiscEval sentence pair classification tasks.
The categories of our benchmark tasks cover a broad range of discourse aspects. The overall accuracies only show a synthetic view of the tasks evaluated DiscEval. Some datasets contain many subcategories that allow for fine grained analysis through wide array of classes (viz. 51 categories for MRDA). Table 5 shows fine grained evaluation which yields some insights on the capabilities of BERT. We report the 5 most frequent classes per task. It is worth noting that the BERT models do not neglect rare classes. These detailed results reveal that BERT+MNLL scores for discourse relation prediction are inflated by good scores predicting the absence of relation (possibly close to the neutral class in NLI), which useful but not sufficient for discourse understanding. The STILTs have complementary strengths even with given tasks, which can explain why combining them is helpful. However, we used a rather simplistic multitask setup, and efficient combination of the tasks remains open problem.
7

---

Under review as a conference paper at ICLR 2020
Persuasiveness Eloquence Relevance Specificity Strength Valence
EmoBank Arousal
Squinky
Dom.
Inf.
Implicature Formality
BERT
75.6
63.5
81.6
78.3
87.1
72.0
69.5
92.2
72.1
98.3
BERT+MNLI
74.7
57.5
82.3
72.2
86.6
72.4
69.9
92.5 73.9
98.1
BERT+DiscEval
75.6
64.0
83,2
82.0
86.8
71.9
69.2
92.3 71.8
98.6
BisTet
73.8
63.0
82.6
79.5
87.1
71.4
70.1
92.6 72.0
97.7
B+DisSent+MNNII
76.9
61.5
83.9
73.9
87.6
72.1
69.4
91.5
73.4
97.9
Discovryyy 76.0
80.1
71.4
72.6
74.2
98.5
B+Discovery+MNIJ
74.1
60.4
79.4
80.4
86.4
72.1
69.6
93.1
75.3
98.4
Human estimate
74.9
73.8
Table 4: Transfer test accuracies across DiscEval subtasks (Persuasivenesss EmoBank, Squinky) BERT+X refers to BERT pretrained classification model after auxiliary finetuning phase on task X.
BERT
B+MNLI B+DisSent B+Discovery B+DiscEval Support
GUM.no o
48.9
51.0
46.0
45.4
43.3
48
SUM.cireumstance
77.1
80.6
73.2
77.8
74.6
35
GUM. elaabrrbor
41.5
38.5
40.0
46.1
42,9
GUM, BUMMbackground
22.6
25.3
34.3
38.2
35.8
23
GUM.evaluation
20.4
22.6
36.8
29.9
35.1
20
STAC.no relation
59.9
63.8
55.4
61.3
46.9
117
STAC.Comment
77.8
76.1
74.9
78.6
54.4
115
STAC. Question answer pair
79.1
80.1
83.3
76.9
83.0
STAC. TTAC.QEElab
32.1
34.3
32.0
38.1
63.7
86
STAC.Contrai
29.6
37.4
25.9
27.5
49.9
53
SwitchBoard. SwitchBoardUuninnepretbbb
86.0
86.0
85.5
86.1
86.3
382
StateeeeeddSSttmeet--eeeee oon-opinion
72.0
72.1
72.4
72.4
72.4
304
SwitchBoard. Ycs-No- Question
Ves-NooQu
85.9
85.2
85.5
85.9
85.8
303
46.3
46.3
48.6
48.8
49.5
113
S SwitchBeard.Apprccee
73.5
71.1
70.2
71.7
72.9
108
PDTB.C PDTB.Cause
55.2
55.7
53.1
57.2
55.9
302
PDTB Restatement
40.4
40.0
41.3
43.9
41.0
263
PDTB C.Bj.nnnnntnoonn
52.8
53.9
52.1
53.3
52.5
262
PDTB Contrast
45.8
49.0
47.2
48.0
46.0
172
PDTB.R IDDBB.Istantiation
56.6
55.6
52.8
58.7
55.7
109
MRDA. Statement
MRDDA.Statement
$1.2
51.8
48.9
53.4
51.4
364
MRDA. Defending//Eplaaaaiio
52.8
54.1
55.3
52.8
52.0
166
MRDA. E.ppasxxanssons of y/n Answers
51.7
48.7
50.3
49.6
49.4
139
MRDA. Offfer
48.6
46.9
50.7
49.4
49.4
102
MRDA. Rising Tone
39.3
40.1
40.3
40.7
38.8
Table 5: Transfer scores across the categories DiscEval tasks; B(ERT)+X denotes BERT pretrained classification model after auxiliary finetuning phase task X.
5 CONCLUSION
We proposed DiscEval, a set of discourse related evaluation tasks, and used them to evaluate BERT finetuned on various auxiliary finetuning tasks. The results lead us rethink the efficiency of mainly using NLI as an auxiliary training task. DiscEval can be used for training or evaluation in general NLU or discourse related work. Much effort has been devoted to NLI for training and evaluation for general purpose sentence understanding, but we just scratched the surface of the use of discourse oriented tasks. In further investigations, we plan to use more general tasks than classification on sentence pairs, such as longer and possibly structured sequences. Several of the datasets we used (MRDA, SwitchBoard, GUM, STAC) already contain such higher level structures. In addition, a more inclusive comparison with human annotators discourse tasks could also help to pinpoint the weaknesses of current models dealing with discourse phenomena. Yet another step would be to study the correlations between performance metrics in deployed NLU systems and scores of the automated
8

---

Under review as a conference paper at ICLR 2020
evaluation benchmarks (GLUE/DiscEval) in order to validate our claims about the centrality of discourse.
REFERENCES
Eneko Agirre, Mona Diab, Daniel Cer, and Aitor Gonzalez-Agirre. Semeval- task 6: A pilot on semantic textual similarity. In Proceedings of the First Joint Conference on Lexical and Computational Semantics- Volume I: Proceedings of the main conference and the shared lask, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation, pp. 385-393. Association for Computational Linguistics, 2012.
James Allen and Mark Core. Draft of damsl: Dialog act markup in several layers, 1997.
Nicholas Asher and Alex Lascarides. Logics of conversation. Cambridge University Press, 2003.
Nicholas Asher, Julie Hunter, Mathieu Morey, Benamara Farah, and Stergos Afantenos, Discourse structure and dialogue acts in multiparty dialogue: the STAC corpus. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC' 16), pp. 2721- 2727, Portorož, Slovenia, May 2016. European Language Resources Association (ELRA). URL
http:::/www.aclleb..rrrggg/ wwwpeb..rgg/tthhoooogoooooooo
6-432
John Langshaw Austin. How to do things with words. Oxford university press, 1975.
Sonia Badene, Kate Thompson, Jean-Pierre Lorré, and Nicholas Asher. Data programming for learning discourse structure. In Proceedings of the 57th Conference of the Association for Com- putational Linguistics, pp. 645, Florence, Italy, July 2019. Association for Computational Linguisties. URL https: www. http://wwww......tttt..ooo/////.
Hongxiao Bai and Hai Zhao. Deep enhanced representation for implicit discourse relation recognition. In Proceedings of the 27th International Conference Computational Linguistics, pp. 571-583, Santa Fe, New Mexico, USA, August 2018. Association for Computational Linguistics. URL
https: www. ac web. org hta/tattoooolloooooooo httpg///aaaoooooo/1111111100
Yonatan Belinkov and James Glass. Analysis methods in neural language processing: A survey. Transactions of the Association for Computational Linguistics, 7:49-72, 2019.
Samuel R Bowman. Modeling natural language semantics in learned representations. PhD thesis, 2016.
Sven Buechel and Udo Hahn. EmoBank: Studying the impact of annotation perspective and representation format dimensional emotion analysis. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pp. 578 585, Valencia, Spain, April 2017. Association for Computational Linguistics. URL www.wcll...cowwwwwwwwwwwwwwww.ww
https:/ // www. ac www.lweeba.c rg antholo SenthooooyyEEr. ogy/E17 /E1/ 20///E172211 2092
Winston Carlile, Nishant Gurrapadi, Zixuan Ke, and Vincent Ng. Give me more feedback: Annotating
argument persuasiveness and related attributes in student essays. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 621 221- Melbourne, Australia, July 2018. Association for Computational Linguistics. URL https://wwwt aww.aaawell org http.///r//atooog///oeeeee..
Daniel Cer. Yinfei Yang, Sheng yi Kong. Nan Hua. Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedler Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. Universal sentence encoder, 2018.
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loie Barrault, and Antoine Bordes. Supervised Learning of Universal Sentence Representations from Natural Language Inference Data. Emnlp, 2017.
Alexis Conneau, Germán Kruszewski, Guillaume Lample, Loic Barrault, and Marco Baroni. What you can cram into a single vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2126-2136. Association for Computational Linguistics, 2018. URL http!
web. hww.://wb..g//aatooooo//oe/eeeee

---

Under review as a conference paper at ICLR 2020
Marie- Cathee-eteernee de Marneffe, Mandy Simons, and Judith Tonhauser. The commitmentbank: Investi- gating projection in naturally occurring discourse. Proceedings of Simn und Bedeutung. 23(2):107- 124, Jul. 2019. doi: 10.1814888 10.18144//ub/20019.2222.2o URL Wolnhttps:0 :ttps:://oos..oo ub uni -nin-ktnttnn Ba/sub/Di / nnex. http://pppp.///...bboaiea/
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties:: Human Language Technologies, Volume I (Long Papers). Association for Computational Linguistics, 2019.
Raymond Dolan, Emotion, cognition, and behavior. science, (9955596)11111-1999 2002.
William Ferreira and Andreas Vlachos. Emergent: novel data-set for stance classification. In HLL-NAACL, 2016.
John J. Godfrey, Edward C. Holliman, and Jane McDaniel. Switchboard: Telephone speech corpus for research and development. In Proceedings of the 1992 IEEE International Conference Acoustics, Speech and Signal Processing Volume 1, ICASSP'92, pp. 517-520, Washington, DC,
USA, 1992. IEEE Computer Society. ISBN 0-7803-0532 URL http: //l::///dll.. acm. org/
ci citatii
on oon. wfwfffmmmi....m...m 895550 895693
Mitchell S. Green. Illocutionary force and semantic content. Linguistics and Philosophy. 23(5):
2000, ISSN 01650157, 15730549. URL http: www.
25001787
H. P. Grice. Logic and conversation. In Peter Cole and Jenry L. Morgan (eds.), Syntax and Semantics: Vol. 3: Speech Acts, pp. 41-58, Academic Press, New York, 1975. URL
www.wucl..o ac
uk uk/ls/stune htt/ss/studypacks/////// http:/ppppppaaaaeeeeeeeeeeeeeeee Cerrrrriieeee -ogiiicee--iiccc pdf
M.A.K. Halliday. An Introduction to Functional Grammar. Edward Arnold Press, Baltimore, 1985.
E. Hovy and E. Maier. Parsimonious or profligate: How many and which discourse structure relations? Technical Report RR-93-373, USC Information Sciences Institute, 1992.
Jamie Kiros and William Chan. InferLite: Simple universal sentence representations from natural lan- guage inference data. In Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 4868-4877 Brussels, Belgium, October-November 2018. Association for Computational Linguistics. URL httph::t//ww/e ac www.www.b.... www.wb..rrg/atthoogg///11
Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Skip-thought vectors. In Advunces neural information processing systems, pp. 3294-3302, 2015.
Shibamouli Lahiri. SQUINKY! A Corpus of Sentence-level Formality, Informativeness, and Implica- ture. CORR, abs/1506.022 02.3066,,36, 2015.
02306
Samuel Läubli, Rico Sennrich, and Martin Volk. Has machine translation achieved human parity? case for document-level evaluation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 4791-4796, Brussels, Belgium, Nctober-November 2018. As- sociation for Computational Linguistics. URL https: hwwp://wwwwwwwb....ccccc wwwp:/wwwwww...tt.oooooooooooooe D18-1512
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Improving multi-task deep neu- ral networks via knowledge distillation for natural language understanding. arXiv preprint 2019.
W. Mann and S. Thompson. Rhetorical structure theory: a theory text organization. Technical report, Information Science Institute, 1987.
Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. The natural language decathlon: Multitask learning as question answering. arXiv preprint arXiv: (000::80007777 08730, 2018.
10

---

Under review as a conference paper at ICLR 2020
Mathieu Morey, Philippe Muller, and Nicholas Asher. How much progress have made on RST discourse parsing? a replication study of recent results on the RST-DT. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 1324, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.
1/653/11/D17-1136 136. [tt.ttts:: / /www.ww.. ac web. web..orgg//annhhooog///111111111
Nikita Nangia and Samuel R. Bowman. Human vs. muppet: A conservative estimate of human performance the GLUE benchmark. In Proceedings of the 57th Annual Meeting of the Associ- ation for Computational Linguistics, pp. 4566 4575, Florence, Italy, July 2019. Association for www.:www.rr.//atooollloooo//
Computational Linguistics. URL https::/www..... ac www.wwe.bb.. org www.o//aattooggg....e..e...e//ee
449
Allen Nie, Erin D. Bennett, and Noah D. Goodman. DisSent: Sentence Representation Learning from Explicit Discourse Relations. pp. 4497-4510. July 2019. doi: 10.18653/v1/P19-1144 URL
https://wwet htt::://wwww.o www.::wwwwww.......... www.worttgggggooo/oooooooooooo/e
1442
Shereen Oraby, Vrindavan Harrison, Lena Reed, Ernesto Hernandez, Ellen Riloff, and Marilyn Walker. Creating and characterizing diverse corpus of sarcasm in dialogue. Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pp. 31-41. Association for Computational Linguistics, 2016. doi: 10.18653/11/W16-6 3604. URL http:
7/aclweb. org/ hothoooooooooooooooooooo /W16 3604
Joonsuk Park and Claire Cardie. Identifying appropriate support for propositions in online user comments. In Proceedings of the first workshop on argumentation mining, pp. 29-38, 2014.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. GloVe: Global Vectors for Word
Representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pp. 1532-1543, 2014. ISSN 10495258. doi: 10.3115/v1/D14-111.
Matthew Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018 Con- ference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume (Long Pupers), pp. 2227-2237, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v1/N188 1202. URL
https: www www.ww.......o www.:aclleee. http:///www.wwww..o///////////.. hatrr///ahoooooggMeeeh11
Jeffrey Pfeffer. Understanding the role of power in decision making. Jay M. Shafritz y J. Steven Ott, Classics of Organization Theory, Wadsworth, pp. 137-154, 1981.
Jason Phang. Thibault Févry, and Samuel R. Bowman. Sentence encoders on stilts: Supplementary training intermediate labeled-da -ata tasks. CoRR, abs/1811.01088, 2018. URL httt:://rrxi
3xq/abs/1811. 01088
Adam Poliak, Aparajita Haldar, Rachel Rudinger, J Edward Hu, Ellie Pavlick, Aaron Steven White, and Benjamin Van Durme, Collecting diverse natural language inference problems for sentence representation evaluation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 67-81, 2018.
Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Miltsakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. The penn discourse treebank 2.0. Bente Maegaard Joseph Mariani Jan Odijk Stelios Piperidis Daniel Tapias Nicoletta Calzolari (Conference Chair), Khalid Choukri (cd.), Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08), Marrakech, Morocco, may 2008. European Language Resources Association (ELRA). ISBN 2-9517408-4-0.
Rashmi Prasad, Katherine Forbes Riley, and Alan Lee. Towards Full Text Shallow Discourse Relation Annotation Experiments with Caras-Paraggaph Implicit Relations in the PDTB. (2009), 2014.
Eugénio Ribeiro, Ricardo Ribeiro, and David Martins de Matos. The influence of context on dialogue act recognition. arXiv preprint arXiv:1506.0m 00839, 2015.
John R Searle, Ferene Kiefer, Manfred Bierwisch, et al. Speech act theory and pragmatics, volume 10. Springer, 1980.
11

---

Under review as a conference paper at ICLR 2020
Elizabeth Shriberg, Raj Dhillon, Sonali Bhagat, Jeremy Ang, and Hannah Carvey. The iesi meeting recorder dialog act (mrda) corpus. In Proceedings of the Sth SIGdial Workshop on Discourse and Dialogue HLT-NAACL 2004, 2004.
Damien Sileo, Tim Van de Cruys, Camille Pradel, and Philippe Muller. Mining discourse markers for unsupervised sentence representation learning. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume 1 (Long Papers). Association for Computational Linguistics, 2019. URL
arxiv. wrg. abs /tabs/1900 1903/ 11850
Richard Socher, Danqi Chen, Christopher Manning, Danqi Chen, and Andrew Ng. Reasoning With Neural Tensor Networks for Knowledge Base Completion. In Neural Information Pro- cessing Systems (2003), pp. 924-934, 2013. URL httpps://nnl..eeeeee wtw.fforrdduuupuuue edu/pubs/
BochhrrChenMannt ngNg NNTS22013 pdf
Sandeep Subramanian, Adam Trischler, Yoshua Bengio, and Christopher J Pal. Learning general purpose distributed sentence representations via large scale multi-task learning. International Conference on Learning Representations, 2018.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: multi-task benchmark and analysis platform for natural language understanding. Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 355, Brussels, Belgium, November 2018. Association for Computational Linguistics. URL https::p///// www. wwwweewwbb............ http::ew....g..taaoeeeee
/W18 5446
Alex Wang, Jan Hula, Patrick Xia, Raghavendra Pappagari, R. Thomas McCoy, Roma Patel, Najoung Kim, lan Tenney, Yinghui Huang, Katherin Yu, Shuning Jin, Berlin Chen, Benjamin Van Durme, Edouard Grave, Ellie Pavlick, and Samuel R. Bowman. Can you tell me how to get past sesame street? sentence-level pretraining beyond language modeling. In ACL 2019, 2019a.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations, 2019b. URL
Alex Wang, Ian F. Tenney, Yada Pruksachatkun, Katherin Yu, Jan Hula, Patrick Xia, Raghu Pap- pagari, Shuning Jin, R. Thomas McCoy, Roma Patel, Yinghui Huang, Jason Phang, Edouard Grave, Haokun Liu, Najoung Kim, Phu Mon Htut, Thibault F' F'evry. Berlin Chen, Nikita Nangia, Anhad Mohananey, Katharina Kann, Shikha Bordia, Nicolas Patry, David Benton, Ellie Pavlick, and Samuel R. Bowman. jiant 1.2: software toolkit for research on general-purpose text understanding models. http:///itt /jittt info/ 2019c.
Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. arXiv preprint 18X.v 2018.
John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. Towards universal paraphrastic sentence embeddings. CoRR, abs//511..... 1.111.811 198899, 2015.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, Volume 1 (Long Papers), pp. 1112-1122, Association for Computational Linguisties, 2018. URLLhttp::.
web. www.boorg/aaee http./orrrr.aatoooo////lll.....
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. Xinet: Generalized autoregressive pretraining for language understanding. 2019.
Amir Zeldes. The GUM corpus: Creating multilayer resources in the classroom. Language Resources and Evaluation, 51(3):581-612 2017. doi: http://dk..oi..oo///000/////1190
12

---

Under review as a conference paper at ICLR 2020
APPENDIX A
dataset
Nde
Niest
Ntabels
labels
PDTB
1204
1085
16
[Cause, Conjunction, Restatement, Condition,Concesiii [Elaboration, No-Relation, Cireumstance, Condition, Concession, ...]
...]
GUM
2.59
248
17
STAC
1247
1304
18
[No-Relation, Question-Answer-Patt Comment, Accnowledecckoooeeeeeeeeeee ...]
MRDA
1630
6459
[Statement, Defending/EEplanntion, Expansions Of Y/N Answers. ...]
S SwitchBoard
2113
649
41
[Uninterpretable, Statement- Coiteettt-oo-Oppae Non-Opinion, Yes-No-Quees LissNoo-ueetoon.....
Emergent
259
259
[For, Observing, Against]
Sarcasm
469
469
Verifiability
634
2424
[Unverifiable, Non-Experiential, Experiential]
Persuasivenes-Spoccfici
[Low, High]
Persuasivenes--lloueeee
91
90
[Low, High]
Desuasiveness-Relevanee
91
90
[Low, High]
Persuasiveness-Strnngie
46
46
[Low, High]
Squinky-Forme Formality
453
452
[Low, High]
Squinky-Inii Informatvees
465
464
[Low, High]
Squinky- Implicctue
465
465
[Low, High]
EmoBank- Arousal
684
683
[Low, High]
EmoBank-Dominance
798
798
[Low, High]
EmoBank- Valence
644
643
[Low, High]
Table 6: Number of labels (Niohots), development examples (Ndev), test examples (Nrest) of DvscEval tasks)
13


Reviews:
[{"review_id": "SyxE6vl0tS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "- Overview: This work presents a benchmark for language understanding centered on understanding pragmatics and discourse, in contrast to existing NLP benchmarks which focus on mostly semantic understanding. The stated contributions are\n    - The DiscEval benchmark, with some estimates of human performance\n    - Baseline performance with state of the art models\n    - Comparisons between commonly used supervised training objectives (NLI) and discourse-oriented objectives\n\n- Review: This work offers a complementary evaluation benchmark for NLU systems to what currently exists in the field, and also offers compelling evidence that the current methods used do not provide good signal for learning the types of phenomena assessed in this benchmark. I recommend (weak) accept.\n    - The paper would benefit a lot from having more in-depth explanation of the tasks, what the classes are, and what the classes mean. It's not always clear from the examples or the labels in Table 1 what exactly is being tested.\n        - Additionally, it'd be nice to get the source of the data for each task.\n    - The rough grouping of tasks in nice.\n    - I understand that estimates of human performance, especially for tasks with a high number of classes, can be tricky to obtain. I do feel that they are especially important to have for this dataset. As you said, some of these tasks rely on context that isn't explicitly provided in the utterance, so I wonder if for some of the tasks, there may be insufficient context in just the provided utterances for humans to perform well.\n        - Additionally, some tasks seem to have quite subjective label definitions. For example, \"eloquence\" and \"specificity\" in Persuasion; \"formality\" and \"informativeness\" in Squinky; etc. Validating that humans are consistent on these tasks seems crucial.\n        - For the estimates you do have, are these expert or lay annotators? For some of these annotations, it seems like you'd need trained annotators.\n    - Have you given any thought to exploitable data artifacts that may occur in these datasets that might be driving the fairly high performance on some of these tasks (e.g. Verifiability)?\n    - I think it's quite interesting that fine-tuning on MNLI doesn't lead to good performance on DiscEval, as MNLI, as the authors point out, is commonly taken to be a useful pretraining task. This discrepancy gives practical weight to the authors' claim that discourse and pragmatic phenomena are not being sufficiently studied or evaluated for in current NLP research, despite the fact that these handling these phenomena will be crucial for NLP systems.\n\n- Things to Improve + Questions + Comments\n    - I would hope that most people in the NLP community would not say that language understanding is a solved problem, but I agree that putting \"universal\" in model names and paper titles is a reach-y thing to do.\n    - Tasks\n        - What is the original citation for STAC?\n    - S4.3, Table 2: \n        - Any reason not to try combining all four pretraining/intermediate training tasks in a single model, or at least more combinations of DiscEval with other things?\n        - Could you comment on the standard deviation of the scores (per task) given that you're averaging 6 runs?\n    - Table 3\n        - \"The best overall unsupervised result is achieved with Discovery STILT\": what does unsupervised mean here? It also doesn't look like BERT+Discovery is the best in any column here.\n    - Tables 4 and 5 feel like a bit of a dump of numbers. It'd be more useful to the readers to extract trends (probably using the groupings and theoretical frameworks introduced earlier). The noting of BERT+MNLI being good at predicting absence of relations is nice.\n    - Typos: There are a noticeable number of typos. Here are some I noted:\n        - The formatting of the task descriptions in Section 3 is a bit inconsistent and awkward.\n        - Table 1: \"exemple\"\n        - P4: \"We use a custom train/dev validation split\"\n        - P5: \"that help *realize* speech acts' intentions.\"\n        - P6: \"Prediction of discourse markers based *off* the context clauses...\"\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "SkeKZ0WjqH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a new benchmark for natural language understanding called DiscEval. The benchmark focuses on datasets that more directly measure a model's understanding of the discourse structure and relations in the text. The paper makes direct comparisons to GLUE and especially studies the previous claims of MNLI being a generically good pre-training task. They find that MNLI pretraining, using STILTS, does not help BERT's performance on DiscEval.\n\nSome comments and recommendations,\n- You say that for PDTB you \"select the level 2 relations as categories,\" I believe these are the class level relations. Maybe add in a brief explanation in the paper?\n- Was discarding all but the Strength subclass from the Persuasion dataset and empirically motivated decision or just something you did a-priori?\n- The human results are already hedged, but maybe the grain of salt should be bigger: it occurs to me that the comparison of the model performance and human performance could be unfair since the human performance is reported on the original tasks, before the filtering of data and changing label distribution like with PDTB and GUM.\n- Particularly given the data filtering and restructuring of some of the tasks, getting a rough estimate of human performance for all the datasets would be quite valuable\n- I think saying the MNLI does not help model performance on DiscEval is completely valid but claiming it hurts performance could be a stretch given the margins of error.\n- I know tasks like PDTB and GUM have quite a few classes. Please include the number of classes for each of the datasets in the benchmark. Also including information on the size of the dev and test sets could be helpful.\n\nOverall, I agree with the author's claim that we need to have a broader evaluation suite for NLU, and this benchmark is a step in the right direction. Some of the DiscEval datasets measure NLU information that is somewhat orthogonal to what datasets in GLUE/SuperGLUE measure. I think this will be a useful resource to the research community\n\n\nMinor things,\n- Page 5, last line the CoLA citation is wrong, should be Warstadt et al. (2019)", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "H1xiUtOFFS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThe paper presents a new GLUE-like dataset collection called DiscEval, which focuses on discourse and pragmatics. Like GLUE and SuperGLUE, datasets from existing works are collated and formated. The tasks are all classification tasks.\n\nThe paper also evaluates several baselines including bag-of-words, BiLSTM encodings, and BERT fine-tuned on different types of data, which have shown success in GLUE tasks.\n\nLike other NLP benchmarks, the DiscEval benchmark would be a good resource for other researchers to hill-climb their systems on, provided that the data format is standardized and the submission system is easy to use like GLUE.\n\nThat said, the paper has some rooms for improvement:\n\n- With the information in Table 2, it is hard to judge the difficulty and headroom for each task. Only a few tasks have human evaluation scores were estimated from the inter-annotator agreement. Contrast this to the GLUE and SuperGLUE papers which provide human baselines from actual humans. Without these anchoring numbers, it is hard to see if the remaining gap is due to the model's inability to model discourse, or due to noise in the dataset.\n\n- Providing the best single-task result from previous work would also help give a more complete picture.\n\n- With the result of fine-tuned BERT almost matching the human performance in several tasks, the argument that BERT is not a universal representation (abstract + introduction) is weakened somewhat.\n\nAs a valuable resource for other researchers, I am still leaning toward acceptance despite the issues above.\n\nOther comments:\n\n- The bullet points on Page 5 could be clarified. Currently, the first bullet seems to contain multiple groups, for example.\n\n- Sentiment analysis (in GLUE) could be viewed as a discourse task. It would be nice to be a bit more upfront about it.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new benchmark to evaluate natural language processing models on discourse-related tasks based on existing datasets that are not available in other benchmarks (SentEval/GLUE/SuperGLUE). The authors also provide a set of baselines based on BERT, ELMo, and others; and estimates of human performance for some tasks.

I think this has the potential to be a valuable resource to the research community, but I am not sure that it is the best fit for a conference such as ICLR. R3 also raises a valid concern regarding the performance of fine-tuned BERT that are comparable to human estimates on half of the tasks (3 out of 5), which slightly weakens the main motivation of having this new benchmark. 

My main suggestion to the authors is to have a very solid motivation for the new benchmark, including the reason of inclusion for each of the tasks. I believe that this is important to encourage the community to adopt it. For something like this, it would be nice (although not necessary) to have a clean website for submission as well. I believe that someone who proposes a new benchmark needs to do as best as they can to make it easy for other people to use it.

Due to the above issues and space constraint, I recommend to reject the paper.
Decision: Reject