Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents the GuoFeng benchmark consisting of i) 9 NLP tasks, ii) a contrastive testing set, iii) a large scale training data. The aim is to evaluate the abilities of models for cohesion and coherence across clauses and sentences. The authors also conduct experiments with transformer models using various settings, including randomly initialized ones, pre-trained models, and pre-trained models on the GuoFeng training data. 
strength_and_weaknesses: Strengths:

* The datasets are valuable for the document-level NLP research, which need to address the coherence and cohesion across clauses and sentences.
* The experiments provide baseline results for future research.

Weaknesses:
* I have difficulties to fully understand the settings of the 9 tasks. How to evaluate each task?
     * SI: are the speakers always mentioned before the utterance of interest? What are the expected outputs if there are multiple utterances with quotation marks?
     * ZPR: What are Chinese-English movie subtitles? Are they Chinese movies with English subtitles, English movies with Chinese subtitles, or both?
     * NT: Is eclipsis not important for novel translation? How about concept consistency in scientice fictions?
     * CCT: do you mean traditional Chinese? Novels written in traditional Chinese?
     * PT: what's the target language? English?
     * How to evaluate the generation tasks if there are multiple plausible answers?
* What is the inter-annotator agreements for the tasks requiring human annotations?
* What are the inter-annotator agreements for the discourse-aware test suite? How are the annotators instructed to construct adversarial examples or the noises are added by algorithms? What kind of noises?
* Why coreference resolution models are not considered as part of or as baselines of applicable tasks? 
* It is not clear to me what are the new challenges arising from the benchmark? 
clarity,_quality,_novelty_and_reproducibility: * Clarity is certainly an issue, see the weaknesses.

* Reproducibility is also an issue due to the lack of human annotation details and the guidelines for human annotators.
summary_of_the_review: Overall, this paper can benefit from a major revision, although the resources are interesting for the community. More insights from the experiments are expected, not just model A is better than model B, expecially about what are the new challenges that are not covered by the prior works, evidenced by the experimental results.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper describes a new collection of datasets, the goal being to analyze discourse phenomena.
strength_and_weaknesses: Strengths: data set construction is valuable, and the goals of the work seem worthwhile.

Weaknesses: first, there is no attempt to evaluate the quality of the resulting data, either through expert evaluations or inter-annotator consistency (I think the former is the only way to really check for this). Second, many of the details of the paper are unclear. Third, many claims are made without sufficent support. 
clarity,_quality,_novelty_and_reproducibility: Reproducibility is problematic given that no quality control/measurement is described.

Clarity is an issue in several places.
summary_of_the_review: Overall, this is worthwhile work, but there are a few issues:

* the lack of any measures of quality of these datasets is problematic. I would be hesitant to use this data without such measures

* Clarity is an issue in several places. what does "aims to recover omitted pronouns in terms of position and form, according to its anaphora information in the given sentence" mean?, for example

* I think the argument that these are "discourse level" tasks is a stretch in several cases. For example, several of the datasets are conventional translation datasets.

Overall I think the goal to release data is valuable, but the paper needs to be improved.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a novel set of benchmark resources for discourse-aware evaluation of language undersatnding, translation, and generation, focusing on Chinese and English languages. The release includes benchmarking data, diagnostic test suite, pretraining corpora and pretrained models. The benchmark is applied on relevant models across a wide variety of tasks, showing benefits of document-level pretraining in Transformer architectures when it comes to discourse.
strength_and_weaknesses: Strengths:
- Potentially a very useful resource for discourse-aware evaluation across tasks.
- Very thorough description of the resources, including in-depth background in the appendix.
- Considerable experiment breadth.
- High-quality writeup.

Weaknesses:
- The paper does not reflect thoroughly on the linguistic bias introduced by choosing Chinese and English as the focus languages. There should be a Limitations section that addresses how the proposed approach would scale in low-resource scenarios or if extending the breadth of linguistic coverage in the datasets. What would it take to introduce another language? What if that language is low-resource? How would that support the findings? Can we simulate that with the existing data to an extent?
- Experiment breadth is not backed with experiment depth, at least in the main part of the paper, as the Main Results and Analysis are rather superficial. This is a minor concern since the main focus of the paper is to provide the resource.
clarity,_quality,_novelty_and_reproducibility: The paper is exceptionally clear, of high quality, and while it is a resource paper so it does not contribute novel modelling, it does offer a very thorough account of discourse in natural language understanding, translation, and generation.
summary_of_the_review: Contained above.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The benchmark, especially the dataset offered in this paper can be a good resource for document-level NLP tasks, which has not been the main focus in ML conferences. Having said that I also agree with Reviewer CWMG that it should do more to ensure the dataset quality, presentation clarity and baselines. Although some of these have been addressed in the rebuttal but the reviewers feel the paper is not yet ready to be published at its current form.  
justification_for_why_not_higher_score: Not good enough
justification_for_why_not_lower_score: NA
Decision: Reject