PRET: Prerequisite-Enriched Terminology. A Case Study on Educational Texts
Authors: Chiara Alzetta, Forsina Koceva, Samuele Passalacqua, Ilaria Torre, Giovanni Adorni (Università di Genova, 2018)
Topic: Lexical and Semantic Resources and Analysis

Abstract
In this paper we present PRET, a English. In this paper we present PRET, a gold dataset annotated for prerequisite relations between educational concepts extracted from a computer science textbook, and we describe the language and domain independent approach for the creation of the resource. Additionally, we have created an annotation tool to support, validate and analyze the annotation.

Introduction
Educational Concept Maps (ECM) are acyclic graphs which formally represent a domain’s knowledge and make explicit the pedagogical dependency relations between concepts (Adorni and Koceva, 2016). A concept, in an ECM, is an atomic piece of knowledge of the subject domain. From a pedagogical point of view, the most important dependency relation between concepts is the prerequisite relation, that explicits which concepts a student has to learn before moving to the next. Several approaches have been proposed to extract prerequisite relations from various educational sources (Vuong et al., 2011; Yang et al., 2015; Gordon et al., 2016; Wang et al., 2016; Liang et al., 2017; Liang et al., 2018; Adorni et al., 2018). Textbooks in particular are a valuable resource for this task since they are designed to support the learning process respecting the prerequisite relation. In the literature, the evaluation of the extracted prerequisite relations is usually performed through comparison with a gold standard produced by human subjects that annotate relations between concepts (see, among the others, (Talukdar and Cohen, 2012; Liang et al., 2015; Fabbri et al., 2018)). However, most of the evaluations lack a systematic approach or simply lack the details that allow them to be repeated. In this paper, we present our experience in building PRET (Prerequisite-Enriched Terminology), a gold dataset annotated with the prerequisite relation between pairs of concepts. The issues emerged with PRET led us to deﬁne a methodology and a tool for manual prerequisite annotation. The goal of the tool is to support the creation of gold datasets for validating automatic extraction of prerequisites. Both the PRET dataset and the tool are available online1. PRET was constructed in two main steps: ﬁrst we exploited computational linguistics methods to extract relevant terms from a textbook2, then we asked humans to manually identify and annotate the prerequisite relations between educational concepts. Since the terminology creation step was extensively described in Adorni et al. (2018), this paper mainly focuses on the annotation phase. The annotation task consists in making explicit the prerequisite relations between two distinct concepts if the relation is somehow inferable from the text in question. We represent a concept as a domain–speciﬁc term denoting domain entities expressed by either single nominal terms (e.g. internet, network, software) or complex nominal structures with modiﬁers (e.g. malicious software, trojan horse, HyperText Document). Figure 1 shows 1http://teldh.dibris.unige.it/pret 2For the annotation we used chapter 4 of the computer science textbook “Computer Science: An Overview” (Brookshear and Brylow, 2015).

Conclusion
In this paper, we described PRET, a gold dataset manually annotated for prerequisite relations between pairs of concepts; moreover we presented the methodology we adopted and a tool to support prerequisite annotation. The case study, even limited as for the number of annotators and the educational material, was a reasonably good training ground to set the basis to deﬁne a methodology for prerequisite annotation and to identify the major issues related to this task. Moreover, the analysis of the annotation provided insights for automatic identiﬁcation of concepts and prerequisites, that will be investigated in future work.