Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper studies `"explainable NLP” and surveys available datasets and taxonomizes existing data collection methodologies. Focusing on a categorization of "highlights", "free-form", and "structured" explanations, the paper outlines how these categories appear in the datasets and certain weaknesses (eg artifacts introduced by annotators) or strengths that appear in certain datasets). There is also some high-level prescriptive guidance on best practices to carry forward. 


====
Post author response: 
Thanks to the authors for making the changes! I think they help on clarity. 
I would keep the rating the same because as the authors mention some of the weaknesses are fairly inherent to the abridged survey form. I think the paper could be accepted. 
strengths: The paper is comprehensive and ambitious in scope. I found the paper useful as an overview of datasets with explanations in NLP, though admittedly as an “outsider” to the field so I cannot compare this meta-analysis with other meta-analyses of the same topic. I think this paper is a good resource for navigating the landscape of explainable NLP for outsiders. I found it especially helpful when the paper synthesizes or draws direct comparisons across datasets. 
weaknesses: Overall the paper is a bit of a high-level "meta-analysis" summary, summarizing some meta-data of the datasets and pulling in/drawing out some specific attributes of specific datasets. I find the latter -- synthesizing discussions across datasets, comparing structural qualities across datasets, much more informative than the table by itself. The prescriptive guidelines and reflections in this paper are a bit high-level/descriptive and could be sharpened further. Although I could see this being a good resource for newcomers (Eg like a well-written blog post, for visibility); the sharpness of the analysis seems like there is room for improvement. 

Although the taxonomy is based on formal structure of the explanation, e.g. splitting into highlight / free-form / structured, the paper remains fairly descriptive in summarizing or recounting the metadata of the datasets without finer-grained analysis. At times the editorial analysis (eg section 5) felt a bit informal or under-argued. While it was helpful to include specific examples, a more structured comparison of datasets along the evaluative criteria (eg sufficiency and so on for highlights) would have been helpful. 

I also found it helpful when the paper recounted critiques of datasets, as this may be more evident to “insiders” to the literature than outside. Making this more structured (even including a column for possible concerns in the table --- though this could be  understandably a burden for the authors to maintain) could sharpen the utility of the summarizing tables. 
correctness: Correct to the best of my knowledge.
clarity: The paper is reasonably well written although clarity lessens in the second half/last few sections. 
relation to prior work: It is somewhat discussed how this work differs from previous meta-analysis of explanations in ML and/or NLP. 
additional feedback: 
Minor writing comments: 
Section 6.3: lines 300-303, it would have been helpful to flesh this out with more details; what kinds of tasks? 
Lines 319-325: this argument seems a bit weak to me or self-evident, could somewhat be said more succinctly. What would be more effective than this paragraph, for example, would be including specific examples of the previously mentioned artifacts --- what should dataset creators watch out for specifically that could be introduced by not having enough annotators? 

Typos: line 93, coarsest
review_title: Review

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This survey paper provides a concise overview on 61 datasets for explainable natural language processing, highlighting their advantages/disadvantages and making suggestions for better data collection strategies. It provides a good starting point for EXNLP. The authors start with explaining why textual explanation is important for ML models, and then introduce three types of explanations and existing datasets. In particular they discuss how the dataset collection methods can affect modelling, focusing on the question of whether human explanation is sufficient or comprehensive for predictions (unsurprisingly, it is not). The paper further discuss means for improving the data quality and diversity, for instance, by using template-based explanations.

===Post author response===: Thanks for the responses clarifying the structure of the draft and incorporating the feedback in the final version. The final score remains unchanged as I already thought it can be published.
strengths: A quite comprehensive and well-organised introduction and summary on large amounts of datasets for explainable NLP, which will be handy for both experts and beginners.

The suggestions for data collection provided can guide researchers for creating high quality data.

weaknesses: It is not very clear how to assess explanations in a dataset. Although the authors dedicate some sections for the related discussion, the discussion covers varies domains and dimensions, making it less focused. A short description in section 4 or 6 summing up key takeaways would be great. For instance, what qualifies a good or acceptable explanation? Is there a quantifiable measure? 
correctness: To my knowledge, the claims are solid.
clarity: The paper is well written. However, it appears to me that section 4 to section 6 overall cover similar concept that is the data quality for EXNLP, making the perception that somehow the key messages are scattered here and there. It is a bit difficult to draw a high level picture from them.
relation to prior work: The distinctions between the submission and prior work are clearly made in section 1: the datasets are not addressed before and the authors want to connect the dots between the dataset and modeling experiments.
additional feedback: N/A
review_title: A comprehensive survey

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: This takes a much-needed lens to the details of how ExNLP datasets are actually collected, providing a corrective to the model-centric approach of much of machine learning. It points out some weaknesses in how we collect ExNLP data and how we document the collection process. The suggestions it makes for improvement are reasonable.

===Post author response===: Thanks for making the improvements. My rating is unchanged.
strengths:  - Makes novel points that are key to doing ExNLP properly
 - Provides constructive suggestions for improving current practices
 - Provides many starting points for future work on ExNLP
weaknesses: The paper is a little unfocused and comes across at some points as a collection of different points related to the same subject (datasets). It feels like some sections could be expanded into standalone papers in themselves. However, perhaps this is understandable in a paper that is tackling this subject (dataset quality in ExNLP) for the first time. I think more narrative connecting the different points would help the reader understand any ways in which the different points made contribute to a unified message.
correctness: To my knowledge, the paper makes correct claims.
clarity: I found section 4 difficult to understand. In particular, the claim in Section 4 that human highlights are "independent" of the evaluation of faithfulness was difficult to understand without reading the whole section several times. It would help to explain:
- what do you mean by 'independence' here?
- who makes the independence assumption?
- what are the negative consequences of making the independence assumption?

I also found Table 5 in the appendix to be crucial to understanding this section. I think this table should be in the main text rather than the appendix.
relation to prior work: Yes, it is clear that this is the first paper to question dataset annotation quality when it comes to ExNLP in particular. Other papers have started examining dataset quality in ML in general, but this is the first to focus on dataset quality in ExNLP.
additional feedback: N/A
review_title: Makes good points; could be better organized/written

Meta-review: The paper surveys existing datasets and data collection methodologies for explainable NLP, pointing out strengths and weaknesses of existing datasets and providing suggestions for future data collection in this area. Overall the reviewers found the paper interesting and useful to the community, but had some issues with the clarity and presentation. The authors were largely able to address these concerns with their responses and revised draft, and in the end all reviewers thought that the paper should be accepted. Congratulations on having your paper accepted to the NeurIPS 2021 Track on Datasets and Benchmarks! When preparing the camera-ready paper the authors are encouraged to take the reviewers feedback into account, specifically with improving the clarity and readability of the paper.
Decision: Accept