dataesr/scientific-paragraphs-categorization huggingface dataset:
, 2020a) : -Open QA, where systems collect evidences and answers across several sources such as Web pages and knowledge bases  (Fader et al. , 2013)  -Reading Comprehension  (RC) , where the answer is gathered from a single document. This is the most common approach. RC systems can be oriented to: -Extractive RC, where systems extract spans of text with the answer. This approach has received a lot of attention fostered by the availability of popular benchmarks such as SQuAD  (Rajpurkar et al. , 2018) , NewsQA  (Trischler et al. , 2017)  or TriviaQA  (Joshi et al. , 2017). Each of these datasets contains thousands of examples, which permits to train Deep Learning systems and obtain good results. -Multiple-choice RC, where systems must select an answer from a set of candidates. Multiple-Choice (MC) is a common way to measure reading comprehension in humans.
This is why some researches have pointed MC as a better format to test language understanding of automatic systems  (Rogers et al. , 2020a). There exists several MC collections, mostly in English. In some cases it involves paying crowd-workers to gather documents and/or pose questions regarding those documents. MCTest  (Richardson et al. , 2013) , for example, proposed for the workers to invent short, children friendly, fictional stories and four questions with four answers each, including deliberately wrong answers. As a way to encourage a deeper understanding of texts, the QuAIL dataset includes unanswerable questions  (Rogers et al. , 2020b). Other datasets were created from real world exams. This is the case of the well known MC dataset RACE  (Lai et al. , 2017) , or the multilingual Entrance Exams  (Rodrigo et al.
, 2018). -Generative QA, where systems create a text that answers the question. The exact text is not necessarily contained in any document, which makes this a challenging task. This kind of systems has received less attention given that it is difficult to perform an exact evaluation and there are few datasets available  (Kočiský et al. , 2018). There is a large amount of research in the area of single-turn QA and there are several surveys, we refer the reader to: Kolomiyets and Moens (2011);  Diefenbach et al. (2018) ;  Mishra and Jain (2016). In this survey, we focus on the evaluation of multi-turn QA systems, which is a much less researched area. Context QA. Context QA refers to systems which allow for follow-up questions to resolve ambiguities or keeping track of a sequence of inference steps  (Peñas et al.
This measures the overall experiences of the subjects. Their evaluation showed that the Task Questionnaire is the most effective at distinguishing among different systems. The evaluation of dialogue QA systems requires one to simulate some interactions with users and evaluate them. These interactions can, on the one hand, be created by real users, which is associated with high costs, and makes it hard to reproduce the experiments and reuse the data. For example,  Li et al. (2017c)  developed DailyDialog, a multi-turn dataset with 13k dialogues created by humans that also include emotion information. On the other hand, the interactions can be automatically produced. However, it is challenging to create users' responses automatically. One approach for creating simulations is to provide some feedback based on the supplementary questions.
In fact, there are many features involved in deciding the next response in a dialogue. This has been defined as the one-tomany problem of dialogues  (Zhao et al. , 2017). Automatic evaluations based on multiple-reference responses have been proposed to alleviate the one-to-many problem. Multi-reference based evaluations include several correct responses for a given context. Thus, these evaluations promote diversity better than singleresponse approaches. Sordoni et al. (2015)  created a synthetic multiple-reference dialogue corpus based on Twitter. Additional responses to the initial response were searched using Information Retrieval and rated by crowd workers. The authors kept only responses with a high rate. Galley et al. (2015)  created a dataset from Twitter following the work from  Sordoni et al.
Evaluation Datasets and Challenges Datasets play an important role for the evaluation of dialogue systems, together with challenges open to public participation. A large number of datasets have been used and made publicly available for the evaluation of dialogue systems in the last decades, but the coverage across dialogue components and evaluation methods (e. g. Sections 3 and 4) is uneven. Also note that datasets are not restricted to specific evaluation methods, as they can be used to feed more than one evaluation method or metric interchangeably. In this section, we cover the most relevant datasets and challenges, starting with select datasets. For further references, refer to a broad survey of publicly available datasets that have already been used to build and evaluate dialogue systems carried out by  Serban et al.
(2018). The dialogue datasets selected for this section are listed in Tables 9, 10 and 11, where properties such as the topics covered and number of dialogues are indicated. 6. 1 Datasets for Task-Oriented Dialogue Systems Datasets are usually designed to evaluate specific dialogue components, and very few public datasets are able to evaluate an entire task-oriented dialogue system (e. g. Section 3). The evaluation of these kinds of systems is highly system-specific, and it is therefore difficult to reuse the dataset with other systems. Their evaluation also requires considerable human effort, as the involvement of individual users or external evaluators is usually needed. For example, in  Gasic et al. (2013) , which is a Partially observable Markov decision processbased dialogue system mentioned in Section 3.
3. 1 for the restaurants domain, the evaluation of policies is done by crowd-sourcers via the Amazon Mechanical Turk service. Mechanical Turk users were asked first to find some specific restaurants, and after each dialogue was   2017 ) used them to build and evaluate a whole dialogue system, re-adjusting the dataset by ignoring the state annotation and reusing only the transcripts of dialogues. The Schema Guided Dialogue (SGD) dataset released for the 8th edition of DSTC was designed to test not only state tracking, but also intent prediction, slot filling and language generation for large-scale virtual assistants. SGD consists of almost 23K annotated multi-domain (bank, media, calendar, travel, weather,. ), task-oriented dialogues between a human and a virtual assistant. The MultiWOZ (Multi-Domain Wizard-of-Oz) dataset represented a significant breakthrough in the scarcity of dialogues as it contains around 10K dialogues, which is at least one order of magnitude larger than any structured corpus available before  (Budzianowski et al.
, 2018). It is annotated with dialogue belief states and dialogue actions, so it can be used for the development of individual components of a dialogue system. But its considerable size makes it very appropriate for the training of end-to-end based dialogue systems. The main topic of the dialogues is tourism, containing seven domains, such as attractions, hospitals, police, hotels, restaurants, taxis and trains. Each dialogue can contain more than one of these domains. Similar in size and content to MultiWOZ is Taskmaster-1 task-based dialogue dataset  (Byrne et al. , 2019). It includes around 13K dialogues in six domains: ordering pizza, setting auto repair appointments, arranging taxi services, ordering movie tickets, ordering coffee drinks and making restaurant reservations. What makes it different from the previous one is that more than a half of the dialogues are created following a self-dialogue methodology, in which a crowd-worker writes the full dialogue themselves.
The authors claim that these  self-dialogues have richer and more diverse language than, for example, MultiWOZ, as it is not restricted to a small knowledge base. The largest human-generated and multi-domain dialogue dataset that is available to the public is MultiDoGo  (Peskov et al. , 2019) , which comprises over 81K dialogues. These dialogues were created following the Wizard-of-Oz approach between a crowd-worker and a trained annotator. These participants were guided to introduce specific biases like intent or slot change, multi-intent, multiple slot values, slot overfilling and slot deletion in conversations. Additionally, over 54K of the total amount of the dialogues are annotated at the turn level for intent classes and slot labels. Dialogues are from six different domains: airline, fast food, finance, insurance, media and software support.
We will conclude this section by discussing two related tools, rather than a dialogue dataset. The first tool, called PyDial 11 , partially addresses the shortage of evaluation datasets for task-oriented systems. This is because it offers the opportunity for developing a dialogue management environment, based on reinforcement-learning for benchmarking purposes  (Ultes et al. , 2017). Thus, it makes it possible to evaluate and compare different task-oriented dialogue systems in the same conditions. This toolkit not only provides domain-independent implementations of different modules in a dialogue system, but also simulates users (see Section 3. 4. 2). It uses two metrics for the evaluation: (1) the average success rate and (2) the average reward for each evaluated policy model of reinforcement-learning algorithms.
The success rate is defined as the percentage of dialogues that are completed successfully. Thus, it is closely related to the task-completion metric used by the PARADISE framework (see Section 3. 4. 1). Another dialogue annotation tool is called LIDA  (Collins et al. , 2019). The authors argue that the quality of a dataset has a significant effect on the quality of a dialogue system, hence, a good dialogue annotation tool is essential to create the best annotated dialogue dataset. LIDA is the first annotation tool that handles the entire dialogue annotation pipeline from turn and dialogue segmentation through to labelling structured conversation data. Moreover, it also includes an interface for inter-annotator disagreements resolution.
Datasets for Conversational Dialogue Systems Regarding the evaluation of Conversational dialogue systems presented in Section 4, datasets derived from conversations on micro-blogging or social media websites (e. g. Twitter or Reddit) are good candidates, as they contain general-purpose or non-task-oriented conversations that are orders of magnitude larger than other dialogue datasets used before. For instance, Switchboard  (Godfrey et al. , 1992)  (telephone conversations on pre-specified topics), British National Corpus  (Leech, 1993)  (British dialogues many contexts, from formal business or government meetings to radio shows and phone-ins) and SubTle Corpus (Ameixa and Coheur, 2013) (aligned interaction-response pairs from movie subtitles) are three datasets released earlier that have 2,400, 854 and 3.
This is labeled by crowdsourced annotators, who measure the quality of a response in a given context  (Sordoni et al. , 2015). Alternatively,  Lowe et al. (2015)  hypothesized that chat-room style messaging is more closely correlated to human-to-human dialogues than micro-blogging websites like Twitter, or forum-based sites such as Reddit. Thus, they presented the above-mentioned Ubuntu Dialogue Corpus. This large-scale corpus targets a specific domain. Thus, it could accordingly be used as a task-oriented dataset for the research and evaluation of dialogue state trackers. However, it also has the unstructured nature of interactions from microblog services that makes it appropriate for the evaluation of non-task-oriented dialogue systems. These two large datasets are adequate for the three subtypes of non-task-oriented dialogue systems: unsupervised, trained and utterance selection metrics.
Notice that, additionally, some human judgments could be needed in some cases, such as in  Lowe et al. (2017a)  for the ADEM system (see Section 4. 3. 1). Here, they use human judgments collected via Amazon Mechanical Turk in addition to the evaluation using the Twitter dataset. Apart from the afore-mentioned two datasets, the five datasets generated recently for bAbI tasks  (Bordes et al. , 2017)  are appropriate for evaluation using the next utterance classification method (see Section 4. 3. 2). These tasks were designed for testing end-to-end dialogue systems in the restaurant domain, but they check whether the systems can predict the appropriate utterances among a fixed set of candidates, and are not useful for systems that generate the utterance directly. The ibAbI dataset mentioned in the next section has been created based on bAbI to cover several representative multi-turn QA tasks.
Another interesting resource is the ParlAI framework 15  for dialogue research, as it contains many popular datasets available all in one place with the goal of sharing, training and evaluating dialogue models across many tasks  (Miller et al. , 2017). Some of the dialogue datasets that are included have been already mentioned (bAbI Dialog tasks and the Ubuntu Dialog Corpus) but it also contains conversations mined from OpenSubtitles 16  and Cornell Movie 17.
Datasets for Question Answering Dialogue Systems With respect to QA dialogue systems, two datasets have been created based on human interactions from technical chats or forums. The first one is the Ubuntu Dialogue Corpus, containing almost one million multi-turn dialogues extracted from the Ubuntu chat logs, which was used to receive technical support for various Ubuntu-related problems  (Lowe et al. , 2015). Similarly, MSDialog contains dialogues from a forum dedicated to Microsoft products. MSDialog also contains the user intent of each interaction  (Qu et al. , 2018). ibAbI represents another approach for creating multi-turn QA datasets  (Li et al. , 2017a). ibAbI interactivity adds to the bAbI dataset that was previously presented (see Section 6. 2) by adding sentences and ambiguous questions with the corresponding disambiguation question, which should be asked by an automatic system.
The authors evaluate their system regarding the successful tasks. However, it is unclear how to evaluate a system if it produces a modified version of the disambiguation question. Recently, several datasets that are very relevant for the context of QA dialogue systems have been released. The CoQA (Conversational Question Answering) dataset contains 8K dialogues and 127K conversation turns  (Reddy et al. , 2018). The answers from CoQA are free-form text with their corresponding evidence highlighted in the passage. It is a multidomain dataset, as the passages are selected from several sources, covering seven different domains: children's stories, literature, middle and high school English exams, news, articles from Wikipedia, science and discussions from Reddit. QuAC (Question Answering in Context) consists of 14K information-seeking QA dialogues (100K total QA pairs) over sections from Wikipedia articles about people  (Choi et al.
, 2018). What makes it different from other datasets so far is that some of the questions are unanswerable and that context is needed in order to answer some of the questions. Another similar dataset that has unanswerable questions and its questions are context-dependent is DoQA, a dataset for accessing domain-specific Frequently Asked Question sites via conversational QA  (Campos et al. , 2019). It contains 1,637 information-seeking dialogues on the cooking domain (7,329 questions in total). An analysis carried out by the authors showed that in this dataset there are less factoid questions than in the others, as DoQA focuses on open-ended questions about specific topics. Amazon Mechanical Turk was used to collect the dialogues for the three datasets.
Evaluation Challenges We conclude this section by summarizing some of the recent evaluation challenges that are popular for benchmarking state-of-the-art dialogue systems. They have an important role in the evaluation of dialogue systems, not only because they offer a good benchmark scenario to test and compare the systems on a common platform, but also because they often release the dialogue datasets for later evaluation. Perhaps one of the most popular challenges is the Dialog State Tracking Challenge (DSTC)  18  , which was previously mentioned in this section. DSTC was started in 2013 in order to provide a common testbed for the task of dialogue state tracking. It continued on a yearly basis with remarkable success. For its sixth edition, it was renamed as Dialog System Technology Challenges due to the interest of the research community in a wider variety of dialoguerelated problems.
Various well-known datasets have been produced and released for every edition: DSTC1 has human-computer dialogues in the bus timetable domain; DSTC2 and DSTC3 used human-computer dialogues in the restaurant information domain; DSTC4 dialogues were human-human and in the tourist information domain; DSTC5 also is from the tourist information domain, but training dialogues are provided in one language and test dialogues are in a different language. Finally, as the DSTC6 edition consisted of 3 parallel tracks, different datasets were released for each track, such as, a transaction dialogue dataset for the restaurant domain, two datasets that are part of OpenSubtitles and Twitter datasets, and different chat-oriented dialogue datasets with dialogue breakdown annotations in Japanese and English.
A more recent challenge that started in 2017 and continued into 2018, with its second edition being the Conversational Intelligence Challenge (ConvAI)  19. This challenge, conducted under the scope of NIPS, has the aim to unify the community around the task of building systems capable of intelligent conversations. In its first edition teams were expected to submit dialogue systems able to carry out intelligent and natural conversations about specific news articles with humans. The aim of the task of the second edition has been to model normal conversation when two interlocutors meet for the first time, and get to know each other. The dataset of this task consists of 10,981 dialogues with 164,356 utterances, and it is available in the ParlAI framework mentioned above. Finally, the Alexa Prize 20 has attracted mass media and research attention alike.
Wen et al. (2017)-oriented dialogue systems. finished,theyhad to fill in a feedback form to indicate if the dialogue had been successful or not. Similarly, for the end-to-end dialogue system byWen et al. (2017)(cf. Section 3. 3. 2),alsofor the restaurants domain, human evaluation was conducted by users recruited via Amazon Mechanical Turk. Each evaluator had to follow a given task and to rate the system's performance. More specifically, they had to grade the subjective success rate, the perceived comprehension ability and naturalness of the responses. Most of the task-oriented datasets are designed to evaluate components of dialogue systems. For example, several datasets have been released through different editions of the Dialog State Tracking Challenge 10 , focused on the development and evaluation of the dialogue state tracker component.
Funding This work was supported by the  LIHLITH  project in the framework of  EU ERA-Net CHIST-ERA ; the  Swiss National Science Foundation  [ 20CH21 174237 ]; the  Spanish Research Agency  [ PCIN-2017-11 ,  PCIN-2017-085/AEI ];  Eneko Agirre  and  Arantxa Otegi  received the support of the  UPV/EHU  [grant  GIU16/16 ];  Agence Nationale pour la Recherche  [ ANR-17-CHR2-0001-03 ].
However, even if these datasets were designed to test state tracking,Bordes et al. ( Name Topics # dialogues Reference DSTC1 Bus schedules 15,000 (Williams et al. , 2013) DSTC2 Restaurants 3,000 (Henderson et al. , 2014) DSTC3 Tourist information 2,265 (Henderson et al. , 2013a) DSTC4 & DSTC5 Tourist information 35 (Kim et al. , 2016) DSTC6 Restaurant reservation - (Perez et al. , 2017) DSTC7 (Flex Data) Student guiding 500 (Gunasekara et al. , 2019) DSTC8 (MetaLWOz) 47 domains 37,884 (Lee et al. , 2019) DSTC8 (Schema-Guided) 20 domains 22,825 (Rastogi et al. , 2019) MultiWOZ Tourist information 10,438 (Budzianowski et al. , 2018) Taskmaster-1 6 domains 13,215 (Byrne et al. , 2019) MultiDoGo 6 domains 86,698 (Peskov et al. , 2019).
Original research added and as new injuries and injury outcomes are incorporated. 11 Comprehensive research of this nature shows how injury burden varies dynamically by age, sex, year, area of the world and type of injury, and hence, it is important to maintain close monitoring of injury burden every year in all parts of the world. In addition, as new datasets and statistical modelling methods become available, producing regular updates to burden estimation also ensures that results are as accurate as possible. While the burden of injuries is widely studied and monitored through various methods of research, the Global Burden of Diseases, Injuries, and Risk Factors (GBD) Study is the only study framework that routinely provides estimates of morbidity and mortality from an exhaustive list of injuries in all areas of the world across ages and sexes.
3] [14] [15] [16] [17]  As part of this regular update, new datasets on cause of death and incidence are incorporated into the study, and additional geographical detail is added to better measure heterogeneity in burden estimates at a subnational level. In addition, updates such as reporting both nature of injury and cause of injury (described in more detail below) are incorporated. In this study, we describe key components in the GBD injury methodology and provide results from key trends in injury burden in terms of incidence, prevalence, years lived with disability (YLDs), cause-specific mortality, years of life lost (YLLs) and disability-adjusted life years (DALYs) by country, age groups, sex, year and injury type.
Cause-specific mortality estimation Cause-specific mortality from injuries is measured using the Cause of Death Ensemble model (CODEm). CODEm is described in more detail elsewhere; a summary of its use for injuries is as follows. 18 First, all available data that can be used for cause of death estimation are identified. For injuries, this includes vital registration, verbal autopsy, police records, mortuary data and census data. These data are processed for use in the GBD cause and demographic hierarchy via a series of data processing steps including a process whereby ill-defined causes of death are reassigned to true underlying causes of death, which is described in more detail elsewhere but essentially is the process by which illdefined causes of death are reclassified to causes of death in the GBD cause hierarchy.
19 20 Next, a cause-specific mortality model is developed for each one of the 30 different causes of injury. For example, falls are modelled differently than road injuries, though both use the same CODEm modelling architecture. For each cause of injury, covariates that may be associated with the cause are identified and added as candidate covariates. CODEm runs different combinations of models using different covariates and outcome variables, specifically cause fraction models and causespecific mortality rate models. Ensembles of models are also conducted to test performance of overall models formed from submodels. Once all models have been run, the top-performing models are selected based on out-of-sample predictive validity, wherein the model makes predictions on data that were not included in developing the model.