DARC-IT: a DAtaset for Reading Comprehension in ITalian
Authors: Dominique Brunato, Martina Valeriani, Felice Dell'Orletta (Università di Pisa, CNR-ILC, 2018)
Topic: Text Simplification

Abstract
In this paper, we present DARCEnglish. In this paper, we present DARCIT, a new reading comprehension dataset for the Italian language aimed at identifying ‘question-worthy’ sentences, i.e. sentences in a text which contain information that is worth asking a question about1. The purpose of the corpus is twofold: to investigate the linguistic proﬁle of questionworthy sentences and to support the development of automatic question generation systems.

Introduction
Reading comprehension (RC) can be deﬁned as “the process of simultaneously extracting and constructing meaning through interaction and involvement with written language” (Snow, 2002). Such a deﬁnition emphasizes that RC is a complex human ability that can be decomposed into multiple operations, such as coreference resolution, understanding discourse relations, commonsense reasoning 1The corpus will be made publicly available for research purposes at the following link: http://www.italianlp.it/resources/ 2Il corpus sar`a messo a disposizione liberamente per scopi di ricerca al seguente indirizzo: http://www.italianlp.it/resources/ and reasoning across multiple sentences. In educational scenarios, student’s comprehension and reasoning skills are typically assessed through a variety of tasks, going from prediction tasks (e.g. cloze test) to retellings generation and question answering, which are costly to produce and require domain expert knowledge. Given also the challenges posed by the broad diffusion of distance learning programs, such as MOOC (Massive Open Online Courses), the automatic assessment of RC is becoming a rapidly growing research ﬁeld of Natural Language Processing (NLP). While much more work has been done on developing Automated Essay Scoring (AES) systems (Passonneau et al., 2017), recent studies have focused on the automatic generation of questions to be used for evaluating humans’ reading and comprehension (Du and Cardie, 2017; Afzal and Mitkov, 2014). This is not a trivial task, since it assumes the ability to understand which concepts in a text are most relevant, where relevance can be here deﬁned as the likelihood of a passage to be worth asking a question about. The availability of large and high-quality RC datasets containing questions posed by humans on a given text thus becomes a fundamental requirement to train data-driven systems able to automatically learn what makes a passage ‘question-worthy’. In this regard, datasets collected for other NLP tasks, Question Answering above all, provide a valuable resource. One of the most widely used is the Stanford Question Answering Dataset (SQuAD), (Rajpurkar et al., 2016). It contains more than 100,000 questions posed by crowdworkers on a set of Wikipedia articles, in which the answer to each question is a segment of text from the corresponding reading passage. More recently, other large RC datasets have been released: it is the case of the ‘TriviaQA’ dataset (Joshi et al., 2017), which is intended to be more challenging than SQuaD since it contains a higher proportion of complex questions, i.e. questions requiring inference over multiple sentences. The same holds for RACE (Lai et al., 2017), which is also the only one speciﬁcally designed for educational purposes. Indeed it covers multiple domains and written styles and contains questions generated by domain experts, i.e. English teachers, to assess reading and comprehension skills of L2 learners. While all these datasets are available for the English language, to our knowledge, no similar RC datasets exist for the Italian language. In this paper we introduce a new corpus for Italian speciﬁcally conceived to support research on the automatic identiﬁcation of question-worthy passages. In what follows, we ﬁrst describe the typology of texts it contains and the annotation process we performed on them. We then carry out a qualitative analysis based on linguistic features automatically extracted from texts with the aim of studying, on the one hand, which features mostly discriminate question-worthy sentences from other sentences and, on the other hand, whether the two classes of sentences have a different proﬁle in terms of linguistic complexity.

Conclusion
We presented DARC-IT, a new reading comprehension dataset for Italian collected from a sample of standardized evaluation tests used to assess students’ reading and comprehension at different grade levels. For each text, we annotated ‘question-worthy’ sentences, i.e. sentences which contained the answer to a given question. A qualitative analysis of these sentences showed that the likelihood of a sentence to be ‘questionworthy’ can be modeled using a set of linguistic features, which are especially linked to syntactic complexity. We believe that this corpus can support research on the development of automatic question generation systems as well as question answering systems. Current developments go into several directions: we are carrying out a ﬁrst classiﬁcation experiment to automatically predict ‘question-worthy’ sentences and evaluate the impact of linguistic features on the classiﬁer performance. We are also planning to enlarge the corpus and to investigate more in-depth the typology of questions and answers it contains, in order to study what characterizes sentences answering, for instance, to factual vs non-factual questions.