Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: Recommendation: 2: Serious ethical issues that need to be addressed in the final version

Ethics Review:

The authors introduce the TGEA 2.0 dataset which is a Chinese dataset where the examples are generated by various pretrained language models. The dataset has been annotated such that the machine-authored texts can be assessed on various tasks within the broad categories of diagnosis tasks and pathology mitigation tasks.

The main issue raised by reviewers is the risk of erasure and invisibility of linguistic variability in Chinese language training data. A recommendation was formulated in this regard.
Ethics Documentation:


strengths: To address the reviewers' concerns on erasure of specificities in the Chinese language, the authors offered to contact the developers of the publicly available models they are using with the idea of asking for information on the training data used for these models. The authors also offered to include data cards for th emodels and datasets, especially with respect to the varieties of Chinese. The goal is to identify varieties of Chinese other than Mandarin. More generally, authors propose to provide a clear description with respect to the variety of Chinese in the revised version of the paper.

weaknesses: N/A
correctness: N/A
clarity: N/A
relation to prior work: N/A
additional feedback: N/A
review_title: This is not a review. My ethics review has been done for this paper.

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper presents a large-scale and curated dataset in Chinese along with two benchmarks for diagnosis (5 tasks) and pathology mitigation (2 tasks) to improve quality of generated texts from language models. The authors designed a thorough annotation process including data collection, training annotators in the pre-annotation phase and quality control by the feedback loop. The selected sentences for annotation cover 3 aspects: model, decoding strategy and prompt. They also provided a detailed analysis on the distributions of erroneous sentences produced by a variety of models with different sizes ranging from 110M to 2.6B parameters. The experimental results on the proposed benchmarks show that the diagnosis tasks are challenging and training a language model (here, GPT-2) on their dataset help reducing errors in the generated texts.
strengths: 1. Data collection phase is well-implemented: The authors use different models, decoding strategies and prompt types (nominal, phrasal and sentential) with various domains (News, Wikipedia and Web Fictions) to diversify types of erroneous sentences.
2. The annotation process and quality control are well-designed to annotate the large-scale dataset while maintaining its quality.
3. The dataset has potential usages: Large language models in Chinese can benefit from training on this dataset to mitigating erroneous sentences. Also, the discriminative models can be trained to automatically detect errors made by language models. Thus, this work can be valuable for future research.
weaknesses: 1. I have some concerns regarding the quality control:
* L188: Who trained the first 4 reviewers? I would like to evaluate the quality of this dataset carefully, since their annotations are used as ground-truths to train other annotators.
* L199: What is average performance of 7 well-trained reviewers? Are they trained by the annotations produced by the first 4 reviewers? The reason I asked is that they are the ones who guarantee the high-quality outcomes for this dataset.
2. Although alpha-balanced loss was used, some diagnostic tasks such as Erroneous Text Detection, MiSEW Extraction suffer a heavy unbalance that may affect model training and evaluation, so the results on these datasets are not quite convincing.
3. No statistics reported for the proposed tasks in two sets of benchmarks.
4. The Word Prediction task in the pathology mitigation benchmark does not properly evaluate the ability of language models because there can be many correct predictions for the last token given each sentence.
5. No qualitative examples (i.e., model predictions) for each benchmark task in the main text and the appendix.
6. Some minor issues in presentation:
* Numbers in Table 1 are quite small that makes it hard to read
* L240: Error Correction task is missing.
* L267: It should be MacBERT instead of MacBEERT.
correctness: Overall, the dataset is well-constructed given the carefully designed data collection and annotation processes, especially the quality control. The proposed benchmarks derived from the dataset have some flaws and need to be improved for better evaluation of language models.
clarity: Although there are some minor issues (see weaknesses), the paper is quite clear and well written in general.
relation to prior work: The paper provides a detailed discussion about the prior work and also points out the differences so I think they have a solid contribution.
additional feedback: There are two suggestions for improvements:
1. Provide human performance to see the upperbound of proposed benchmark tasks.
2. Train the other 3 models used in data collection phase (i.e., NEZHA-Gen, PanGu-alpha, CPM) and ask annotators/reviewers to evaluate their generated texts to see if training on the dataset helps them avoid making errors.

and two questions regarding data collection:
1. L148: How to obtain 20K paragraphs from 170K prompts? Did you filter out any unqualified paragraphs? 
2. L151: With only the first sentence selected from 20K paragraphs (thus, 20K sentences in total), I am not quite sure how to obtain 195,629 annotated sentences.
review_title: Well-constructed Dataset for Language Models

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The authors introduce the TGEA 2.0 dataset which is a Chinese dataset where the examples are generated by various pretrained language models. The dataset has been annotated such that the machine-authored texts can be assessed on various tasks within the broad categories of diagnosis tasks and pathology mitigation tasks.
strengths: The main strength of this dataset is its scale: it substantially extends TGEA 1.0 to now consist of 195,629 annotated sentences. Such a dataset will be particularly useful in devising methods to assess the quality of the generated text from pre-trained language models. Also, the authors have taken great care for a sophisticated quality control process in order to ensure that the annotations for the various benchmarking tasks can be trusted.
weaknesses: Further explanation or clarification is required for the following points:

-	The authors claim that the examples generated are diverse due to the different decoding strategies and also using 4 different pretrained language models. However, are 4 pretrained language models representative in the mistakes they make for future pretrained language models that have come out recently and to come out which are far larger in size and may have different pathological weaknesses?

-	It would be great if the novelty in the tasks beyond TGEA 1.0 could be clearly spelled out beyond just the scale of the dataset.

-	How valid is it to compare the pathological weaknesses of models in Chinese to English examples as in SCARECROW?

correctness: The dataset seems to have been constructed sensibly and with a lot of care.
clarity: Yes – very clear to read, but a small typo in line 32 (the grammar seems a bit off).
relation to prior work: Yes, the authors outline how TGEA 2.0 builds on top of TGEA 1.0 and SCARECROW but it would be good to further highlight the differences between TGEA 2.0 and TGEA 1.0.
additional feedback: Additionally, the results section offers a large number of values for various systems. It would be good if further insights could be provided beyond simply stating the values on why certain baseline systems do better than other systems for each of the tasks.
review_title: Valuable large-scale pretrained language model authored text assessment data and tasks.

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper proposes TGEA 2.0, the largest dataset for diagnosing typed errors made by pretrained language models. It is an extended version of TGEA, with various large language models and downstream tasks. The paper mainly compared its contribution with TGEA and ScareCrow. Several experiments are performed using the dataset, and experimental results on various downstream tasks show that there are large rooms exploring the proposed dataset.
strengths: - It nicely expands previous TGEA in terms of scalability, annotation richness, etc.
- Strict quality control on the construction process
- Proposed MiSEW and pathology mitigation which can assist the annotation richness.
weaknesses: - The intention of MiSEW extraction is plausible, but is somewhat overlapped with erroneous span location as a downstream task. Thus the necessity of the task should be more justified in some manner, such as qualitative analysis.
- Proposed pathology mitigation should also be more explained in terms of why it should be jointly considered in future works.
correctness: The authors properly designed annotation processes and quality control.
clarity: The paper is well-written.
relation to prior work: It clearly compared with previous works (TGEA and ScareCrow).
additional feedback: 1. Minor correction
- "w/ 1 es" in Table 1: 3,1097 → 31,097

2. It would be better to discuss the future direction of class imbalance between correct/errneous texts.
review_title: Review of TGEA 2.0

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This paper contributes to understanding and reducing the text generation errors made by large pre-trained language models. The authors have created the largest Chinese language dataset of machine-authored texts and a substantial subset of the texts (>195k) were manually annotated at a fine-grained level and corrected for text quality issues (grammaticality and semantic coherence). The authors use the annotated data both to benchmark the best performance of 4 major PLMs (with variable architecture and scale) against each top-level error category and to test the extent to which fine-tuning with the human-corrected texts reduces the prevalence of these errors in the PLM outputs. 
strengths: The size of the dataset and multiple annotations (erroneous spans and minimal set of error-related words and corrections for these words) provide an excellent basis for studying the nature of grammatical and semantic coherence errors in Chinese language automated text generated by current SOTA models performing at their best. As such, this dataset should be of considerable interest to researchers seeking to understand the persistence of certain error types in machine-generated text, and also, to understand, from objective evidence (vs subjective human evaluations, cf. Clark 2021), what kinds of errors may be indicative of machine-generated outputs and therefore may support the detection of machine-generated text. The further interest of this paper is the attempt to use the error-plus-correction MiSEW pairs to improve the quality of the generated text by reducing errors of these kinds in the output. Another strength of the paper is the clear and comprehensive description of the research process, including the annotation process, which adheres to annotation best practices in many ways (including an indicator of annotator confidence, pre-training to support annotator convergence to a high level of inter-annotator agreement and iterative re-training during annotation, etc).
weaknesses: It would have been good to see Cohen's Kappa statistics (or similar) cited for the inter-annotator agreement in section 3.3. Average accuracy of annotators is mentioned ("average performance ... increases from 58.9% to 79.7%") and also "inter-annotator disagreement" but not measure of the latter is explicitly provided. 
correctness: Yes, the claims made in the submission appear correct and the benchmark dataset has been created in an appropriate and correct manner. Substantial care has been given to the impact of variables such as model size, decoding strategy etc on the errors occurring in the generated texts, and analysis of the impact of these variables on the error type frequency and distribution in the dataset.
clarity: Yes, the paper is very well-written and clear. There are a very few typos/grammatical errors which would be good to correct: line 32, add "on" after "done"; line 39, substitute "with which" for "that"; standardise "rational" to "rationale" throughout.
relation to prior work: Yes, the relationship to prior work is sufficiently discussed in the opening sections of the paper.
additional feedback: In the Conclusion section, I would be interested to hear what the authors see as the next step for their work -- whether it would be creating a further dataset or specific lines of exploration of the TGEA 2.0 dataset they have created.
review_title: Large-scale dataset for diagnosis and mitigation of Chinese language text generation errors

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The work builds on by releasing a larger and higher quality version of a previously released dataset named TGEA. It is a comprehensive collection of machine authored texts in Chinese language that have been annotated for errors based on a novel ontology of errors. This ontology is based on data mining for frequently occurring forms of errors followed by supervision by expert annotators. Furthermore, systematic analysis to the annotated errors is performed to reveal patterns that are helpful in gauging the capability of various PLMs on different datasets.

Lastly, the work goes on to validate whether errors found can be fixed automatically with pre-existing large language models and find  it hard to solve by modern means. This paves the way for 1) A dataset to analyze the kind errors PLM makes 2) Developing automated methods that can automatically correct the errors made by PLMs because existing SoTA is not enough to rectify it 3) Benchmark of the performance of SoTA on both diagnostic as well as pathological errors for future works to compare against.


strengths: * Improvement over previous work where authors used the stochastic decoding strategy and repetition penalty which reduced the redundancy related errors frequency and hence also allowed energy to be focused on harder errors.
* The number of annotated samples is large enough to gain confidence and mitigate risk of incorrect conclusions due to spurious correlations.
* Benchmark results are shared on the presented dataset using SoTA models which allows the research community to have solid baselines to compare their research and findings against. Furthermore, by evaluating numerous PLMs on diverse datasets, the work also helps users of PLMs in deciding appropriate model for any given NLG task.

weaknesses: * The reasoning behind choosing three point scale for asking confidence in annotation instead of standard Likert scale is not provided.
* Any inspiration that was taken from related work is missing for the choice of annotator training methodology.
* Given that beyond a certain threshold, PLMs show emergence capabilities. The work done here is called into question as whether or not the errors patterns in small PLMs (<5B parameters) are also prevalent in Large PLMs (> 100B) like PanGu-α (200B), Wu Dao 2.0 (1.75T).


Grammar errors:
1. L168 "think it"
2. L190 "in three times"

correctness: Yes, the datasets are constructed in sound manner along with the appropriate experiment design and evaluation methodology.
clarity: The paper is clear and well written. The authors avoid use of complex jargon and present concepts in a simplified manner making it easy for readers to digest the concepts being proposed.
relation to prior work: The work discussed tangents that it is related to as well as how it improves the shortcomings of existing works. There is one aspect whose relation to prior work is overlooked, methodology used to train the annotators that annotated the tasks. 
additional feedback: Authors, kindly upload your datasets to mindspore as well in due time.
review_title: The authors present an updated version of a dataset in Chinese that is annotated with errors made in NLG tasks by Pretrained Language Models.

Meta-review: metareview: The reviewers all liked the paper. The authors' response clarified most points raised by the reviewers. In view of that, the authors are strongly invited to take the feedback on board for the final version. The main ethical issue raised by reviewers is the risk of erasure and invisibility of linguistic variability in Chinese language training data. Data cards need to be added to the final version.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept