dataesr/scientific-paragraphs-categorization huggingface dataset:
In addition to those challenges, GEM datasets also differ in their communicative goals, languages, the noisiness of data, and resource availability, to evaluate the consistency of evaluation schemes. About half of the datasets have multiple references and more Figure  1 : The opportunities of living benchmarks and pitfalls of evaluation. As models improve, we need consistent evaluations such that models can be compared to each other. This can only happen if we develop robust human evaluation standards and improve our automated metrics. Otherwise, results are challenging to interpret and compare to each other. Finally, as models improve and metrics saturate, we need to evaluate them on more challenging datasets instead of continuing to move sideways on old ones. GEM aims to provide this environment for natural language generation.
than half were post-processed to improve data quality. The sizes range from 5k to 500k data points. GEM features 18 languages across all tasks and two of the datasets do not include English at all. To be able to properly assess the performance of models in a way robust to the shortcuts a model can take, we additionally introduce ten types of challenging test sets that probe for specific modeling aspects  (Perez-Beltrachini and Gardent, 2017; Ribeiro et al. , 2020). To ensure that research with GEM is conducted responsibly, all the datasets are documented in an NLG-specific version of data cards  (Bender and Friedman, 2018; Gebru et al. , 2018)  we developed and for which we release a template and guide. Moreover, all submitted models will have an associated data card  (Mitchell et al. , 2019).
This is especially challenging for benchmarks in NLG since, as discussed above, the performance cannot be described through a single metric and it is often not clear what metric to optimize for. This shortfall can be seen in benchmarks like DecaNLP  (McCann et al. , 2018)  and GLGE  (Liu et al. , 2020a)  which include NLG tasks but focus only on a single metric and, as a result, may mischaracterize a system's performance. Moreover, an easy-to-use data infrastructure also disincentivizes researchers from interacting with and conducting in-depth analyses of the data sets that models are trained on. The limited analysis delegates the responsibility to ensure that all included datasets have been collected fairly to the creators of the benchmark  (Denton et al. , 2020). The dataset and benchmark creators thus must provide in-depth statements that describe the data characteristics and surface potential issues and consider these issues when selecting datasets for a benchmark  (Gebru et al.
, 2018; Bender and Friedman, 2018). These dangers emphasize selecting datasets for a benchmark needs to be carefully done, that the setup has to remain flexible to be able to address newly found limitations, and that the benchmark should focus on climbing a leaderboard. Instead, a living benchmark that can adjust its datasets and specific evaluation metrics can be much more powerful and long-lived. This can, for example, be seen in Dynabench, 1  (Potts et al. , 2020)  which has a static evaluation, but interactively adds more test data through a human-in-the-loop approach. Increasing multilingualism of NLG research. Another potentially harmful choice by benchmark creators is the choice of the languages of the included datasets. It is often assumed that work on English transfers to other languages  (Bender, 2011).
However, this assumption does not consider differences between the languages that lead to higher modeling complexity, for example, a richer morphology or a flexible word-order. Still, the majority of work in NLP and almost all benchmarks exclusively focus on English (e. g. ,  Wang et al. , 2019b; Liu et al. , 2020a; McCann et al. , 2018). Even if multiple languages are considered, the availability of data in a language often does not represent the number of speakers of a language. This means that work on languages with little available data can potentially impact many more people than work on highly resourced languages  (Joshi et al. , 2020). As a result, many recent benchmarking and dataset creation efforts in NLU develop and focus on tasks that are inherently multilingual or which explore cross-lingual transfer.
For example, XTREME  (Hu et al. , 2020)  introduces a benchmark covering 40 languages across multiple NLU and retrieval tasks, XCOPA  (Ponti et al. , 2020)  is a commonsense reasoning dataset for eleven languages, and MLQA  (Lewis et al. , 2020b ) is a dataset for extractive question answering across seven languages. We can observe a similar recent trend in natural language generation, where ML-Sum  (Scialom et al. , 2020)  and WikiLingua  (Ladhak et al. , 2020)  were created as multilingual summarization datasets. There also have been first steps toward including NLG tasks in multilingual NLU benchmarks. For example, XGLUE includes Question and News Title Generation  (Liang et al. , 2020). Unfortunately, XGLUE reduces the generation evaluation to BLEU-4, a metric that is inadequate for NLG  (Reiter, 2018).
There have also been multiple shared tasks in NLG that focus on multilingualism, for instance, the shared task on multilingual surface realization which includes eleven languages  (Mille et al. , 2018 (Mille et al. , , 2019 (Mille et al. , , 2020)). The shared task on document-level 1 https://dynabench. org/ generation and translation featured German and English generation challenges  (Heafield et al. , 2020). The WebNLG+ shared task asked participants to contribute models that can realize text in Russian and English  (Ferreira et al. , 2020). A benchmark that focuses only on NLG can enable much richer evaluation (as described in the next sections), and promote non-English datasets. In addition, it can ensure that the datasets created for those shared tasks continue being evaluated. Providing a testbed for automated evaluation.
, 2018 Ma et al. , , 2019;; Mathur et al. , 2020b). The latest metrics focus on semantic equivalence instead of lexical similarity, which improves the correlations drastically. However, recent work by  Fabbri et al. (2020)  demonstrates that this may not hold in summarization, where the automated metric BERTScore  (Zhang et al. , 2020b)  does not improve upon the correlation of ROUGE. Moreover,  Mathur et al. (2020a)  and  Freitag et al. (2020)  find that when comparing two high-quality systems, differences according to a metric may also stem from how references are written or flaws in the metric itself. 2 iven that automated metrics perform differently across tasks, setups, and languages, a multi-task NLG benchmark has the opportunity to act as a testbed to evaluate how the latest advances in automated metrics perform on these different tasks.
Dataset Selection As highlighted in Figure  1 , the selection of included datasets is an integral part of a benchmark. They should be challenging for models, but it should still be possible to evaluate models trained on them. Moreover, the datasets should cover a wide range of relevant generation challenges that allow for findings to be as general as possible. Finally, the datasets should cover tasks that are interesting for contributors to work on to facilitate the wide adoption of the benchmark. To collect datasets with those desired properties, the selection methodology for GEM is composed of three steps. First, we elicited a set of proposals from everyone involved in the effort. Second, we identified criteria for the selection. Third, all GEM members voted on individual dataset and criteria utilities.
The final selection maximizes the utility under constrained resources, similar to a knapsack solver. 4 This can be seen as an extension of the selection process of SuperGLUE  (Wang et al. , 2019a)  that had similar first and second steps but made the final decision based on which were harder for a baseline model to solve after identifying a final set of candidate datasets. Since we are going to introduce challenge sets, the baseline performance of models on a dataset matters less. Dataset Elicitation. In the first step, all GEM participants were asked to suggest datasets following the schema provided in Appendix A. The categories included multiple brief categorizations, such as a description of the challenge that this dataset provides, its high-level task, and the communicative goal of an agent trained on the data.
Following our goal to focus on non-English languages, we further asked for the languages included in the dataset, as well as the language locale. This step yielded 35 proposed datasets, listed in Appendix B. Estimating Task+Criterion Utility. The second step focused on the selection of criteria to inform the selection. The initial set of criteria was selected through open discussion involving all members. We split criteria into "hard" and "soft" ones -hard criteria would lead to the definite inclusion/exclusion of a task if (not) satisfied. Soft criteria inform the utility of the remaining tasks. All GEM members filled out a survey asking them to rate, on a 5-point Likert scale, how much they wanted to see a task included in GEM. Additionally, we posed yes/no questions for all considered hard criteria and various questions about the soft criteria (e.
g. , "what percentage of the tasks should feature non-English language?", or "do we prefer noisy or clean datasets?"). Finally, the survey included open text fields that asked for (1) comments on any of the tasks, (  2 ) comments or suggestions on hard exclusion criteria, and (3) suggestions of additional criterion/criteria. The full list of questions is shown in Appendix C. The survey received 28 responses, revealing that the initial version of GEM should include a median of 10 tasks or an average of 12. Of those tasks, about a third should feature non-English language. Selected Criteria. For the hard criteria, there was an agreement to focus only on open-access datasets and that concurrent or past shared tasks for the same datasets are not an issue. Overall, the sentiment determined the following selection principles: • We focus on diverse high-level tasks over a single high-level task evaluated in-depth.
However, each high-level task should include multiple datasets. • We focus on clean datasets to avoid conflating model mistakes and learned noise. • We include a mix of high-and low-resource datasets. • We focus on data with interesting test sets. • We should not focus on the quality of current evaluation strategies for a given dataset. • We prefer multi-reference datasets since those have been shown to lead to more robust automatic evaluation. High-Level Tasks. Since these principles dictate that we should focus on a small set of high-level tasks, we used the free-text replies to evaluate the interest in different high-level tasks. Grouping the proposed tasks yielded the following candidates: Summarization, Dialog, Simplification/Compression, Question Answering, Creative Writing, Data-to-Text, and Question Generation.
5  There was a preference to exclude image inputs and question answering because those tasks add complexity to the evaluation beyond the generated text. Moreover, since creative generation tasks like story generation and poetry generation suffer even more from inadequate evaluation approaches, there was 5 For a full overview of potential future expansions and challenges, we refer to the survey by  Gatt and Krahmer (2018). a consensus to not include them. There was, however, a strong preference for the high-level tasks Summarization, Data-to-text, and Dialog. 6 pecific Datasets. The final selection is shown in Table  1. To arrive at the selection, we first ranked all datasets by their average rating. For this, we treated positive ratings as 1, negative ratings as -1, and neutral ratings as 0.
The highestranked datasets were E2E with 0. 577, XSum with 0. 538, and ToTTo with 0. 461. Unfortunately, non-English datasets were ranked lower, with only WebNLG and MLSum among the top 15 datasets. We grouped all datasets by their high-level tasks and selected a group that would not violate the selection principles (e. g. , only high-resource tasks). If two datasets fit, we picked the one with a higher interest rating. Among the 11 datasets, we have 18different languages, and the dataset sizes range from 5,000 examples to 1. 5M, with most datasets between 50-150k examples. Two of them do not include English at all, which we hope reduces the dependence of the modeling approaches on anglocentric pretraining  (Anastasopoulos and Neubig, 2020). The high-level tasks include Dialog, Summarization, Data-to-Text, and Simplification.
GEMifying the data We produce data cards  (Bender and Friedman, 2018; Gebru et al. , 2018)  for all data sets in GEM, for which we developed an NLG-specific template. 7 In addition to describing the data itself, the cards acknowledge potential limitations of a dataset regarding its creation process and describe its real-world use cases to ensure that the research is conducted responsibly. These datasets are the base selection, and as part of GEM, we may change datasets and how they are used. For example, we may improve the training sets, make the test sets more challenging, or probe for specific skills a model must exhibit with testonly datasets  (Perez-Beltrachini and Gardent, 2017; Linzen, 2020; Ribeiro et al. , 2020; Schlegel et al. , 2020). We may also ask to evaluate a single model on multiple test sets, following the design by  Dua et al.
(2019). We are including modifications to several of the datasets: (1) MLSum: We excluded all languages besides Spanish and German since the sources for other languages disallow scraping content. Additionally, we removed all duplicate items (i. e. , items with the same input text) and we used langdetect  8 to filter out examples that were in the wrong language. In total, 147 examples were removed from the German portion (0. 06%) and 7417 examples were removed from the Spanish portion (2. 5%). (  2 ) XSum: Summaries in this dataset often have divergence issues between the source and target texts since gold summaries are introductory sentences prefacing each article. Models agnostic to such noises are vulnerable to hallucinations  (Wiseman et al. , 2017; Dhingra et al. , 2019). To combat this, we fine-tuned a BERT-based  (Devlin et al.
, 2019)  classifier on 500 document and gold summary pairs, manually annotated for faithfulness  (Maynez et al. , 2020)  and excluded all document-summary pairs from the original XSum dataset where the classifier was not confident (p(faithful) > 0. 8) whether the summary is faithful to the document or not. (  3 ) Schema-Guided Dialog: We are focusing on the response-generation part of the dataset and thus reformatted the dataset to treat the service agent utterances as the targets to be generated and the previous customer utterance and the agent's dialog act as the input. We additionally reformat the dialog acts to directly conform to the format described in the paper  (Kale and Rastogi, 2020). (  4 ) Wik-iLingua: We focus on the same five languages that were benchmarked in its original release (en, es, ru, tr, vi) in a cross-lingual setup in which the inputs are in the respective language and the outputs are in English.
g. , subsets with different complexity), and (III) the compilation of new test sets (e. g. , out-of-vocabulary inputs). We restrict the size of each challenge set to about 500 examples to minimize computational overhead. On the WebNLG challenge sets, all subset items are selected proportionally from each category to ensure a similar distribution to the original set; on all other datasets the subset items are selected from the whole set. The results of the different systems on these subsets will be compared to the results obtained by the same systems on the same subsets of the original test data. For case (I), altering existing test sets, the first challenge set adds numerical variation in WebNLG. This variation attempts to respect the format of the current cardinal value (e. g. alpha, integer, or floating-point) and replaces the existing value with a new random value as a means to challenge existing trained models.
The generated number is lower-bounded between zero and upper bounded to be within to the highest power of 10 unit for the given value (e. g. replacing a value of 54 would result in a random value between 0-100). Floating values are also bounded to have the same degree of precision as the input value. For structureto-text and dialog datasets, we produce a version of the test sets in which the order of the components of the input structures (triples, concepts, dialog acts, table rows, etc. ) is randomly changed. For text-to-text datasets and Schema-guided Dialog, we introduce several types of perturbations: (a) typographical errors, using butter-fingers  9    by  Xie et al. (2020). We rejected backtranslation outputs based on a character length to ensure that the difference in character length between original and backtranslation does not exceed 35% of the original source character length.
For XSum 99. 8% of the backtranslations were accepted, for Wiki-Auto 94. 42% (ASSET) and 87. 18% (TURK), and for Schema-Guided Dialog 78%. In case (II), the breaking down existing sets, we first provide for each dataset random samples of training and validation data, in order to assess to what extent the scores of the different systems drop when run on the test data. Then, specific splits are created for particular datasets, in order to assess possible biases of the models, and their robustness across inputs with different specifications. For ToTTo, test set splits are built according to several aspects that can be identified using Wiki-Data: gender, ethnicity and nationality grouped by continent. For gender, we compare the performance between male and female people, but cannot compare other genders due to a lack of original data -only seven people in the original test set are marked as having a different gender.
Across all three challenge subsets, we additionally match the fraction of the existing non-overlap and overlap properties. For WebNLG, we propose subsets based on the shape of the inputs (number of triples, number of common subjects and/or objects, depth, etc. ) For Turk/ASSET, splits are created in terms of the syntactic complexity of the sentences to be simplified. To characterise sentence complexity we use the developmental level scale proposed by  Covington et al. (2006). 10 For all datasets, we propose splits based on the frequency of the parts that compose the input in the training data; the resulting test sets range from being made of very common components to being made only from components unseen in the training data. For case (III), we collect timeshifted test data for news summarization in the form of articles with Covid19-related keywords.
Since MLSum and XSum were collected before the pandemic, we can measure how a model responds to context not seen in the training data (outside of potential pretraining). The new set of articles covers existing article topics (economy, sports, etc. ) but all in relation to the Covid19 pandemic. In addition, some new topics appear in the collected data derived from outlet sections that were not part of the original data collection. 11 ince the GEM test sets and final metrics selection have not been released yet, we describe an experimental setup that will ensure that participating models are trained correctly and evaluated on publicly available data with available metrics that will give a sufficient indication of a model's performance. To do this, we are reporting the results of the baseline models on the validation sets.
Modeling Baselines Much of the recent modeling progress in NLP can be attributed to the rise of the pretrain-then-finetune paradigm which has led to consistently better results. This finding is consistent with human judgments for summarization, as shown by  Fabbri et al. (2020) , among others. However, many of the tasks included in GEM may not benefit from a language model encoder since their input is not natural language. We thus apply a variety of different architectures that vary in size, complexity, and training schema. Our main baselines are T5 with 60M parameters  (Raffel et al. , 2020)  and BART with 139M parameters  (Lewis et al. , 2020a). For non-English datasets, we use their multilingual counterparts mT5 in various sizes  (Xue et al. , 2020)  and mBART  (Liu et al. , 2020b). We additionally train the following baselines on a subset of tasks: TGen (with added language model and lemma tags denoted as TGen+/++)  (Dušek and Jurčíček, 2016b) , an architecture for generation from dialog acts, an LSTM-based Sequence-to-sequence model with attention  (Bahdanau et al.
These metrics can often be gamed, for example, ROUGE can be improved by increased the output length of the model  (Sun et al. , 2019). Moreover, the reliability of these metrics depends on the quality and number of the references  (Mathur et al. , 2020a; Freitag et al. , 2020). However, on a system-level, they still correlate well with human judgments for some tasks  (Reiter, 2018). Semantic Equivalence. More recently, metrics that rely on pretrained language models have shown improved correlations with human judgments on the segment-level. We thus include BERTScore  (Zhang et al. , 2020b) , a metric based on the similarity of sentence embeddings, and BLEURT  (Sellam et al. , 2020) , a metric that is fine-tuned on human ratings. The reported baseline results use RoBERTa-large  (Liu et al. , 2019)  and mBERT  (Devlin et al.
However, the parallel coordinates approach can scale to a much greater number of metrics than a table. Moreover, by using a parallel coordinates plot instead of a table, it is easy to spot patterns that span multiple metrics, systems, or tasks. For example, the highlighted line in Figure  2  uncovers that, for the T5 baseline on ToTTo, the diversity metrics score higher than other systems while scoring lower on reference-based metrics. Since we only have a single baseline for ToTTo, it is unclear whether this difference can be attributed to the dataset or the system but this relationship will be uncovered once we receive submissions. The final system will additionally be able to display the model cards and other related metainformation associated with submissions. It will also be able to show (and compare) exemplary outputs for each test set.
Collecting more multilingual data Many of the initial datasets in GEM are focused on (American or British) English; we see this release as a starting point for the collection of new datasets to improve the inclusiveness of other languages and cultures. From the task point of view, to ensure the longevity of the dataset, we want it to be practical and socially beneficial. Through GEM, we have developed a set of desired criteria for NLG datasets and we aim to apply this knowledge to data collection and actively work toward reducing the disparity in data availability between languages  (Joshi et al. , 2020). To this end, we are focusing on a task that requires content selection, planning, and surface realization along in a grounded scenario. The idea is in the prototyping stage with prospects broadly towards dialog response generation and topic summarization in multiple languages.
Regular updates to the living benchmark To activate the benefits of a living benchmark that is focused on evaluation, we commit to regular updates for GEM. We invite contributions in the form of model outputs, analyses, and metrics at any time and will automatically update the results presented on our website to incorporate them. For the updates to the dataset selection, we want to consider the input of the wider NLG research community. To do so, we will set up a yearly selection process similar to the one described in Section 3. The first update process will be run after the GEM workshop at ACL 2021. To be able to have a robust comparison between different versions of GEM, we will only replace a small subset of datasets at a time.