Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper presents a new benchmark for natural language understanding called DiscEval. The benchmark focuses on datasets that more directly measure a model's understanding of the discourse structure and relations in the text. The paper makes direct comparisons to GLUE and especially studies the previous claims of MNLI being a generically good pre-training task. They find that MNLI pretraining, using STILTS, does not help BERT's performance on DiscEval.

Some comments and recommendations,
- You say that for PDTB you "select the level 2 relations as categories," I believe these are the class level relations. Maybe add in a brief explanation in the paper?
- Was discarding all but the Strength subclass from the Persuasion dataset and empirically motivated decision or just something you did a-priori?
- The human results are already hedged, but maybe the grain of salt should be bigger: it occurs to me that the comparison of the model performance and human performance could be unfair since the human performance is reported on the original tasks, before the filtering of data and changing label distribution like with PDTB and GUM.
- Particularly given the data filtering and restructuring of some of the tasks, getting a rough estimate of human performance for all the datasets would be quite valuable
- I think saying the MNLI does not help model performance on DiscEval is completely valid but claiming it hurts performance could be a stretch given the margins of error.
- I know tasks like PDTB and GUM have quite a few classes. Please include the number of classes for each of the datasets in the benchmark. Also including information on the size of the dev and test sets could be helpful.

Overall, I agree with the author's claim that we need to have a broader evaluation suite for NLU, and this benchmark is a step in the right direction. Some of the DiscEval datasets measure NLU information that is somewhat orthogonal to what datasets in GLUE/SuperGLUE measure. I think this will be a useful resource to the research community


Minor things,
- Page 5, last line the CoLA citation is wrong, should be Warstadt et al. (2019)
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: - Overview: This work presents a benchmark for language understanding centered on understanding pragmatics and discourse, in contrast to existing NLP benchmarks which focus on mostly semantic understanding. The stated contributions are
    - The DiscEval benchmark, with some estimates of human performance
    - Baseline performance with state of the art models
    - Comparisons between commonly used supervised training objectives (NLI) and discourse-oriented objectives

- Review: This work offers a complementary evaluation benchmark for NLU systems to what currently exists in the field, and also offers compelling evidence that the current methods used do not provide good signal for learning the types of phenomena assessed in this benchmark. I recommend (weak) accept.
    - The paper would benefit a lot from having more in-depth explanation of the tasks, what the classes are, and what the classes mean. It's not always clear from the examples or the labels in Table 1 what exactly is being tested.
        - Additionally, it'd be nice to get the source of the data for each task.
    - The rough grouping of tasks in nice.
    - I understand that estimates of human performance, especially for tasks with a high number of classes, can be tricky to obtain. I do feel that they are especially important to have for this dataset. As you said, some of these tasks rely on context that isn't explicitly provided in the utterance, so I wonder if for some of the tasks, there may be insufficient context in just the provided utterances for humans to perform well.
        - Additionally, some tasks seem to have quite subjective label definitions. For example, "eloquence" and "specificity" in Persuasion; "formality" and "informativeness" in Squinky; etc. Validating that humans are consistent on these tasks seems crucial.
        - For the estimates you do have, are these expert or lay annotators? For some of these annotations, it seems like you'd need trained annotators.
    - Have you given any thought to exploitable data artifacts that may occur in these datasets that might be driving the fairly high performance on some of these tasks (e.g. Verifiability)?
    - I think it's quite interesting that fine-tuning on MNLI doesn't lead to good performance on DiscEval, as MNLI, as the authors point out, is commonly taken to be a useful pretraining task. This discrepancy gives practical weight to the authors' claim that discourse and pragmatic phenomena are not being sufficiently studied or evaluated for in current NLP research, despite the fact that these handling these phenomena will be crucial for NLP systems.

- Things to Improve + Questions + Comments
    - I would hope that most people in the NLP community would not say that language understanding is a solved problem, but I agree that putting "universal" in model names and paper titles is a reach-y thing to do.
    - Tasks
        - What is the original citation for STAC?
    - S4.3, Table 2: 
        - Any reason not to try combining all four pretraining/intermediate training tasks in a single model, or at least more combinations of DiscEval with other things?
        - Could you comment on the standard deviation of the scores (per task) given that you're averaging 6 runs?
    - Table 3
        - "The best overall unsupervised result is achieved with Discovery STILT": what does unsupervised mean here? It also doesn't look like BERT+Discovery is the best in any column here.
    - Tables 4 and 5 feel like a bit of a dump of numbers. It'd be more useful to the readers to extract trends (probably using the groupings and theoretical frameworks introduced earlier). The noting of BERT+MNLI being good at predicting absence of relations is nice.
    - Typos: There are a noticeable number of typos. Here are some I noted:
        - The formatting of the task descriptions in Section 3 is a bit inconsistent and awkward.
        - Table 1: "exemple"
        - P4: "We use a custom train/dev validation split"
        - P5: "that help *realize* speech acts' intentions."
        - P6: "Prediction of discourse markers based *off* the context clauses..."

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: 
The paper presents a new GLUE-like dataset collection called DiscEval, which focuses on discourse and pragmatics. Like GLUE and SuperGLUE, datasets from existing works are collated and formated. The tasks are all classification tasks.

The paper also evaluates several baselines including bag-of-words, BiLSTM encodings, and BERT fine-tuned on different types of data, which have shown success in GLUE tasks.

Like other NLP benchmarks, the DiscEval benchmark would be a good resource for other researchers to hill-climb their systems on, provided that the data format is standardized and the submission system is easy to use like GLUE.

That said, the paper has some rooms for improvement:

- With the information in Table 2, it is hard to judge the difficulty and headroom for each task. Only a few tasks have human evaluation scores were estimated from the inter-annotator agreement. Contrast this to the GLUE and SuperGLUE papers which provide human baselines from actual humans. Without these anchoring numbers, it is hard to see if the remaining gap is due to the model's inability to model discourse, or due to noise in the dataset.

- Providing the best single-task result from previous work would also help give a more complete picture.

- With the result of fine-tuned BERT almost matching the human performance in several tasks, the argument that BERT is not a universal representation (abstract + introduction) is weakened somewhat.

As a valuable resource for other researchers, I am still leaning toward acceptance despite the issues above.

Other comments:

- The bullet points on Page 5 could be clarified. Currently, the first bullet seems to contain multiple groups, for example.

- Sentiment analysis (in GLUE) could be viewed as a discourse task. It would be nice to be a bit more upfront about it.
review_title: Official Blind Review #3

Meta-review: This paper proposes a new benchmark to evaluate natural language processing models on discourse-related tasks based on existing datasets that are not available in other benchmarks (SentEval/GLUE/SuperGLUE). The authors also provide a set of baselines based on BERT, ELMo, and others; and estimates of human performance for some tasks.

I think this has the potential to be a valuable resource to the research community, but I am not sure that it is the best fit for a conference such as ICLR. R3 also raises a valid concern regarding the performance of fine-tuned BERT that are comparable to human estimates on half of the tasks (3 out of 5), which slightly weakens the main motivation of having this new benchmark. 

My main suggestion to the authors is to have a very solid motivation for the new benchmark, including the reason of inclusion for each of the tasks. I believe that this is important to encourage the community to adopt it. For something like this, it would be nice (although not necessary) to have a clean website for submission as well. I believe that someone who proposes a new benchmark needs to do as best as they can to make it easy for other people to use it.

Due to the above issues and space constraint, I recommend to reject the paper.
Decision: Reject